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Apparatus (60) and methods for producing and displaying panoramic images. The apparatus includes a plurality of video sensor heads 
(92), each sensor head generating video signals corresponding to a partial image having a respective field of view (100), such that at least 
some of the fields of view include regions of substantial overlap (124) with respective neighboring fields of view, and a union of the fields 
of view substantially covers a scene having an angular extent beyond the field of view of any one of the sensor heads. A processor (40) 
receives and combines the partial images from the sensor heads to produce a panoramic image (231) of the scene. The panoramic image is 
displayed by defining an annular strip (230) and a point of view outside a volume bounded by the strip, and mapping the panoramic image 
onto the annular strip. An image of the strip, as seen from the point of view, is rendered to a video display (50). 
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VTDEQ TELECONFERENCING 

RELATED APPLICATION 

This patent application claims the benefit of U.S. Provisional Patent Application No. 
60/043,243, filed April 16, 1997, which is assigned to the assignee of the present application, 
and whose disclosure is incorporated herein by reference. 

FIELD OF THE INVENTION 
The present invention relates generally to video teleconferencing systems, and 
specifically to methods and apparatus for producing, transmitting and displaying panoramic 
images for use in video teleconferencing. 

BACKGROUND OF THE INVENTION 

Panoramic imaging systems are known in the art. Such systems frequently make use of 
a single camera with panoramic optics, for example, a fisheye lens, or a combination of a lens* 
and a convex mirror, as in the ''OmniCam" system developed at Columbia University, New 
York. Image processing is applied to the images captured by the camera, in order to remove 
the distortion introduced by the optics. These systems generally have poor resolution, due to 
the inherently low resolution of standard video cameras, or require the use of a very expensive, 
high-resolution camera to achieve adequate picture quality. 

Alternatively, panoramic images may be produced by panning a video camera over a 
scene and then aligning and combining multiple images from the pan into a single "mosaic" 
imase. This approach allows only a single, relatively small region of interest (ROI) in the scene 
to be updated in real time (live images), while other portions of the scene, at which the camera 
is not pointing, are "frozen/' Such cameras cannot easily accommodate rapid changes in the 
ROI, due, for example, to difiFerent participants speaking in a teleconference, because of the 
time it takes the camera to pan from one ROI to the other. Computational methods for image 
mosaicking are well known in the art, as described, for example, in an article by Jaillon, et al., 
entitled "Image Mosaicking Applied to Three-Dimensional Surfaces," published in the 
Proceedings of the Twelfth International Conference on Pattern Recognition (lAPR), October, 
1994, pages A253-257, which is incorporated herein by reference. 

U.S. patent 5,187,571, to Braun et al., which is incorporated herein by reference, 
describes a television system for displaying multiple views of a remote location. The system 
includes a sending station, a composing system and a receiving station. The sending station 

1 
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includes a sequence of commonly-synchronized cameras, arranged so that the fields of view of 
the cameras merge contiguously, without substantial overlap between the fields, to forni an 

o 

aggregate field of view, which is shown in preferred embodiments to be a 360 panoramic 
view. The composing station multiplexes the camera signals so that a viewer at the receiving 
5 station can select a subsection of the aggregate field of view to display on a local monitor. 

Braun et al. state that in order for the filelds of view of the cameras to be contiguous, 
the cameras must be arranged so that all of the fields of view have a common vertex. Two 
optical arrangements are proposed for this purpose: A first embodiment is shown in Figs. 2 and 
3 of the patent, in which the cameras are positioned in a common plane, and two folding 
10 mirrors are associated with each camera to suitably direct its field of view, as well as serving as 
baffles, thus preventing substantial overlap between the fields of view of adjacent cameras. A 
second, substantially more compact embodiment is shown in Figs. 4 and 5, in which the 
cameras are positioned side-by-side, and a pyramidal mirror, having one face corresponding to 
each of the cameras, directs the fields of view. In the second embodiment, there is no provision 
15 made for baffling. 

U.S. patent 5,444,478, to Lelong et al., which is also incorporated herein by reference, 
points out several limitations of the system of patent 5,187,571, and describes a method and 
apparatus for image processing meant to overcome these limitations. In accordance with this 
method, the signals fi-om the cameras, referred to as "source images," are digitized and are then 
20 geometrically transformed, or reniapped, to construct a "target image." It is pointed out that at 
the boundaries between the fields of view of neighboring cameras in the system of the '571 
patent, the viewing angle difference between one camera and its neighbor causes great 
distortions of the image. In the method described by Lelong et al., a user selects a "virtual 
camera" angle and, optionally, a zoom. Data fi-om the source images are then transformed to 
25 produce a target image, largely distortion-ft-ee, such as would have been captured by the virtual 
camera. Whenever a new virtual camera angle or zoom is chosen, the transformation is 
accordingly recomputed. Neither Lelong nor Braun suggests how . the target image is to be 
chosen within the aggregate image, or how a panoramic image might be displayed representing 
the aggregate image. 

30 In some video conferencing systems known in the art, a camera is controlled to pan, tilt 

and zoom automatically, so as to capture an image of whichever meeting participant is 
speaking. For example, the "Limelight" speaker locating system, produced by PictureTel 

0 
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Corporatioa, of Andover, Massachusetts, uses a microphone array, adjacent to a video 
teleconferencing camera, to determine the direction of sounds in a conference room and directs 
the camera to pan and tilt toward the sound. The camera can also be directed to zoom, in order 
to capture multiple speakers simultaneously. 

5 
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SUMMARY OF THE INVENTION 



10 



15 



20 



25 



It is an object of some aspects of the present invention to proxade an improved system 
for video teleconferencing. 

It is a further object of some aspects of the present invention to provide improved 
panoramic video imaging apparatus and methods. 

Yet another object of some aspects of the present invention is to provide methods and 
apparatus for conveniently displaying panoramic video images. 

It is still another object of some aspects of the present invention to provide methods and 
apparatus for efficiently compressing and transmitting images captured by the apparatus. 

In some aspects of the present invention, audio signals are received, processed and 
transmitted in conjunction with video signals. 

An additional object of some aspects of the present invention is to provide a user 
interface for video teleconferencing which is flexible and easy to operate. 

In preferred embodiments of the present invention, a video teleconferencing system 
comprises panoramic image capture apparatus and a panoramic image display, as well as user 
controls and a communications interface for sending and receiving teleconference data over 
communication lines. 

In preferred embodiments of the present invention, the image capture apparatus 
comprises a plurality of substantially stationary video sensor heads and optics coupled to the 
sensor heads. The optics direct the field of view of each of the sensor heads in a different 
direction, such that the fields of view of all the sensor heads share a substantially commonly- 
centered entrance pupil, and such that the field of view of each sensor head overlaps in an edge 
region thereof with that of its neighbors. The fields of view of all the sensor heads together 
cover a desired angular spread, preferably having an angular aspect ratio (width/height) of at 
least 2:1, Preferably, the combined fields of view cover at least a 90^ azimuthal spread, more 
preferably at least 180^^, and most preferably a 360^ spread. The apparatus further includes 
image capture circuitry, which commonly controls the plurality of sensor heads and receives 
and combines video signals from the sensor heads to form a panoramic image. 

The term "video" as used in the present patent application and in the claims refers to 
electronic signals generated responsive to and/or representative of an image. As such, the 
terms "video sensor head" and 'Video signal" may be taken to refer to sensor heads and signals 
conforming with common video standards, such as NTSC and PAL, but may also refer to other 
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Standard and non-standard de\aces and signals used in electronic imaging. It will be understood 
by those skilled in the an that the principles of the present invention are equally applicable to 
standard video systems and to non-standard electronic imaging apparatus and methods. 

In preferred embodiments of the present invention, the optics cause the image captured 

5 by each of the sensor heads to be vignetted in the edge regions thereof Preferably, the overlap 
and vignetting are used to improve the quality of the resultant panoramic image and simplify the 
image processing needed to construct the image. More preferably, the optics are designed so 
that in regions of overlap between neighboring fields of view, the intensity of the image 
received by each of the sensor heads is reduced by vignetting according to a predetermined 

10 fimctional form. Additionally or alternatively, image processing is applied in the edge regions 
of the images to adjust the intensity and add the images together as required. 

The video signals generated by the neighboring sensor heads are mutually registered and 
are selectively summed together in each of the overlap regions. Preferably, the sum of the 
signals due to the reduced intensities is generally equal to the signal that would have been 

15 generated by a single sensor head viewing the region without vignetting or overiap, with a 
substantially smooth transition from one field of view to the^next. 

In some preferred embodiments of the present invention, the image capture optics 
comprise a multifaceted mirror, including baffles, which protrude radially outward relative to 
the facets of the mirror, to prevent optical "crosstalk" between the facets. The baffles are 

20 preferably located at the joints between the facets of the mirror or, alternatively or additionally, 
along the midlines of the facets or in any other suitable location. The facets themselves are 
preferably planar, but may alternatively have a suitable curved shape, preferably convex. The 
curved shape may be spherical or may correspond to a surface of revolution of another conic 
section, or may be of any other suitable aspheric shape known in the art. Such curved facets 

25 are useful in reducing the size of the image capture apparatus and in reducing the effects of 
keystone distortion, in the output image therefrom. 

By comparison with such preferred embodiments, in image capture systems known in 
the art using stationary cameras, such as that described in the above-mentioned U.S. Patent 
5,187,571, there is no provision for such baffling, and no account is taken of the overlap of the 

30 neighboring fields of view. In the system of the '571 patent, the neighboring fields of view are 
assumed to be mutually contiguous, i.e., non-overiapping, necessitating substantial post- 
acquisition image processing, as described in the above-mentioned U.S. Patent 5,444,478< 

5 
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There is also no provision made in such systems for the possibility of using multifaceted mirrors 
with curved facets, as described above. 

In some preferred embodiments of the present invention, the image capture circuitry 
collectively controls and, preferably, mutually synchronizes the plurality of sensor heads and 

5 processes the video signals generated thereby to vary certain parameters of the image, 
preferably varying the parameters as a function of azimuthal angle. The circuitry thus optimizes 
the quality of the panoramic image for at least a plurality of the azimuthal angles that the image 
covers, while maintaining a substantially smooth variation of the parameters over the entire 
image. Preferably, the parameters controlled and optimized by the circuitry include the image 

1 0 brightness, contrast, dynamic range and color balance, inter alia. 

Furthermore, in some preferred embodiments of the present invention, the panoramic 
image is mapped onto a substantially cylindrical image surface, centered at a central point of 
view. In other words, each point in a scene that is captured by the image capture apparatus is 
geometrically transformed, i.e., magnified, either optically or by electronic image processing, or 

15 by a combination of optical and electronic image processing, so that the panoramic image is 
equivalent to an image that would be captured by a cylindrical image detector (if such a 
detector and suitable optics were available), likewise centered at the central point of view. 
Preferably, the mapping is such that the geometrical transformation performed by the apparatus 
varies as a function of azimuthal angle, so that all the participants in a video teleconference 

20 captured by the apparatus are shown in the panoramic image at substantially the same size, even 
if not all the participants are equally distant from the apparatus. 

Although such cylindrical mapping may cause mild distortion of the image when viewed 
on a flat display, it has been found that such distortion is not generally noticeable in 
teleconferencing applications. On the other hand, the cylindrical mapping and azimuth- 

25 dependent variable magnification allows smooth panning through the image when it is 
displayed, as described below, without camera movement and without the need to remap the 
image each time the view angle is changed, as is necessitated by systems known in the art, for 
example, as described in the above-mentioned '478 patent. The cylindrical mapping also 
reduces the effect of keystone distortion, which is characteristic of flat projection 

30 transformations. 

In some preferred embodiments of the present invention, there is a microphone array 
associated with the image capture apparatus, and audio signals generated by the array are 

6 
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transmitted to a remote viewing site, along with the panoramic image. Preferably, the array 
comprises a plurality of omnidirectional microphones, each receiving sound in a different 
location or, alternatively, a plurality of directional microphones, each aimed in a different 
direction. A viewer of the panoramic image at the remote viewing site can choose to hear 
5 sound received from a desired direction, generally from the direction of a person of interest 
shown in the image. The audio signals received by the plurality of microphones are combined, 
preferably in a phase-dependent manner, to generate an output audio signal corresponding to 
the sound incident on the microphones from the desired direction. Alternatively or additionally, 
the audio signals from the plurality of microphones may be processed and fed to a plurality of 
10 speakers at the remote viewing site, so as to provide directional, stereophonic sound. 

Further alternatively, as is known in the art, the microphone array may be used to 
determine the direction of sounds impinging thereon, so as to indicate or select automatically 
the person in the image who is speaking. 

Still further additionally or ahernatively, the audio signals may be combined, as 
15 described above, so that the sound impinging on the microphone array from a certain direction 
is muted. Such a muting function may be invoked, for example, by a person participating in a 
video teleconference who wishes to speak without being overheard at the remote viewing site. 
Similarly, a portion of the panoramic image may be frozen, blacked out or dimmed, or replaced 
by a video looping sequence if it is desired that the portion not be seen by participants at the 
20 remote viewing site. 

In a preferred embodiment of the present invention, the image capture apparatus 
produces a stereoscopic panoramic image. Preferably, the plurality of sensor heads comprises a 
plurality of pairs of first and second sensor heads, suitably spaced. Each pair of sensor heads is 
positioned to receive images of a scene from a respective one of the faces of the multifaceted 
25 mirror, as described above. The image capture circuitry combines the video signals from all of 
the first sensor heads to generate a first panoramic image and from all of the second sensor 
heads to generate a second panoramic image. When the first and second panoramic images are 
viewed using a suitable stereoscopic display, as is known in the art, a. stereoscopic image is 
observed, giving the illusion of depth. It will be appreciated that other methods and apparatus 
30 for panoramic image capture, as are known in the art, are generally unsuited to creating 
stereoscopic images, due to mutual obscuration by the at least two cameras required for 
stereoscopy of each other^s field of view. 

7 
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In some preferred embodiments of the present invention, the display comprises a video 
monitor and a display processor, which renders the panoramic image to the monitor. 
Preferably, the cylindrical image surface onto which the panoramic image is mapped is rendered 
to the monitor in the form of an annular strip, such as would be seen generally if the cylindrical 
5 image surface were observed from a point of view outside a volume contained within the 
surface. Preferably, the point of view is taken sufficiently above the equatorial plane of the 
volume, so that the entire surface can be seen and assumes a generally elliptical shape as 
rendered. This rendering scheme allows the entire panoramic image to be observed on an 
ordinary' video monitor, in a manner that maintains an intuitive correspondence between the 

10 actual positions of participants in a video conference room and the positions of the respective 
images of the participants in the image on the monitor. Preferably, the user at the remote 
viewing site can control certain aspects of the panoramic image, for example, the location of 
the point of view from which the image is rendered and the height and eccentricity of the strip. 
The annular strip comprises a first hemiannulus relatively proximate to the point of view 

1 5 outside the volume and a second hemiannulus relatively distant therefrom. Whereas the second 
hemiannulus is rendered substantially as seen from the point of view, the first hemiannulus is 
shown "inside-out," so that the inner surface of the hemiannulus is displayed on the outside of 
the hemiannulus (since the cylindrical image surface has an image only on its inner side, and 
there is no image associated with the outer side). In order to maintain the images of the video 

20 conference participants in their proper spatial order, the images of the participants in the first 
hemiannulus are preferably left-right reversed. 

As described above, because the panoramic image is mapped onto the generally 
cylindrical surface, the point of view from which the image is rendered can be changed without 
the need for remapping. Changing the azimuth of the point of view is substantially equivalent 

25 to rotating the surface about a central axis thereof Thus, in some preferred embodiments of 
the present invention, a viewer observing the display may select a region of interest, specifically 
of a panicipant of interest, shown in the image, and the cylindrical surface is then "rotated" on 
the monitor, so that the region of interest occupies a central position in one of the hemiannuli. 
Alternatively, such rotation may be engendered responsive to the direction of sound received by 

30 a microphone array associated wth the image capture apparatus, as described above. 
Preferably, an enlarged image of the region of interest is shown in a picture-in-picture (PIP) 
window in the display. Alternatively, the region of interest within the annulus is enlarged. 
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Further preferably, the viewer may control the tilt and zoom of the PIP window display, as well 
as other aspects of the image. 

In the context of the present patent application and in the claims, the terms "annulus" 
and ''annular" refer broadly to any ring-like geometrical figure, not necessarily round. In 
5 preferred embodiments of the present invention, as described above, the figure is round, 
corresponding to the cylindrical surface onto which the panoramic image is preferably mapped. 
It will be appreciated, however, that within the scope of the present invention, panoramic 
images may also be rendered using ring-like figures of other shapes, for example, oblong or 
oval. 

10 Although in the preferred embodiments described above, certain image processing 

functions are associated with the image capture apparatus and others with the image display, it 
will be understood that in other preferred embodiments, these functions may be distributed 
differently between image capture and display terminals. For example, in some preferred 
embodiments of the present invention, the image of the annular strip may be rendered by the 

1 5 image capture circuitry, and then transmitted to substantially any video monitor in the form of a 
standard video signal, rather than as a panoramic image, as described above. In some of these 
preferred embodiments, in which there are certain user controls associated with the video 
monitor, for example, for selecting a region of interest at the remote videoconference location, 
the image capture circuitry generates an on-screen user interface, which is transmitted and 

20 displayed as part of the standard video signal, alongside the panoramic image. A viewer of the 
image can use the user controls to interact with the user interface and convey instructions or 
responses to the remote image capture circuitry. 

It will be appreciated that the principles of the present invention enable a remote viewer 
of a video conference to select a desired region to view and hear or, additionally or 

25 alternatively, to select a viewpoint fi-om which to watch the conference, without the necessity 
of moving any of the sensor heads or altering any of the controls of the image capture 
apparatus. Furthermore, in order to conserve transmission bandwidth between the image 
capture apparatus and the remote display, the apparatus may be controlled to transmit only 
images generated by the sensor head that is directed toward the region selected by the viewer, 

30 or toward the participant who is speaking at a given moment. The remainder of the panoramic 
image is transmitted and refi*eshed on the display only intermittently. 

9 
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In other preferred embodiments of the present invention, on the other hand, the image 
capture apparatus conveys raw video data to the image display system, which then remaps the 
data to the generally cylindrical surface and generates the panoramic image. In such preferred 
embodiments, the image capture apparatus preferably comprises an array of stationary sensor 

5 heads, as described above. Alternatively, however, the image capture apparatus may comprise 
a camera having pan and, optionally, tilt and zoom functions, which are remotely-controlled 
from the image display system, as is known in the art. The display system controls the camera 
to acquire a panoramic sequence of images, which it then processes, remaps as a panoramic 
image, and renders to the monitor in the form of an annular strip, as described above. The 

10 display system may also perform digital zoom functions, to display enlarged images, preferably 
under viewer control. 

In some preferred embodiments of the present invention, the image capture compresses 
image data for transmission to the display system, using image compression methods known in 
the an, for example, in accordance with the JPEG and/or MPEG standards. Preferably, the 
1 5 circuitry controls the sensor heads and transmits image capture parameters so as to minimize 
the data volume of the compressed signals and/or to optimize the quality of the images when 
decompressed and viewed on the display. 

In another preferred embodiment, the circuitry identifies areas of the'image occupied by 
participants in the video conference, and other areas in which only a background is seen. As 
20 long as the background remains unchanged, only the portions of the image corresponding to the 
participants are then transmitted to the display processor, which then receives and updates only 
these portions of the image that is rendered to the monitor. 

Those skilled in the art will appreciate that although preferred embodiments of the 
present invention are described herein with reference to a complete video teleconferencing 
25 system, elements of the system, such as the panoramic image capture apparatus and the 
panoramic image display, may also be used independently of one another, in other 
teleconferencing systems and in other image capture and display applications. 

It will be further appreciated that although preferred embodiments are described herein 
with reference to teleconferencing, the principles of the present invention may similarly be 
30 applied to create, transmit and view panoramic images for other purposes. 

There is therefore provided, in accordance with a preferred embodiment of the present 
invention, apparatus for producing a panoramic image, including: 

10 
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a plurality of video sensor heads, each sensor head generating video signals 
corresponding to a panial image having a respective field of view, such that at least some of the 
fields of view include regions of substantial overlap with respective neighboring fields of view, 
and a union of the fields of view substantially covers a scene having an angular extent beyond 
5 the field of view of any one of the sensor heads; and 

a processor, which receives and combines the partial images fi-om the sensor heads to 
produce a panoramic image of the scene. 

Preferably, the panoramic image covers an azimuthal angle of at least 90**, more 
preferably at least 180^ and most preferably substantially equal to 360**, 
10 Preferably, the panoramic image has an aspect ratio of width to height of at least 2:1. 

Preferably, the sensor heads are substantially stationary. 

Preferably, the panoramic image is mapped onto a generally cylindrical geometrical 
image surface, and the processor renders the image surface to display the panoramic image in a 
generally elliptical form on a video screen. Further preferably, the processor is controlled so as 
15 to geometrically rotate the image surface. 

Preferably, the panoramic image includes a reference image, and wherein the processor 
updates a portion of the reference image responsive to a video signal received ft*om a selected 
one or more of the plurality of sensor heads. 

In a preferred embodiment, the apparatus includes a microphone array, which receives 
20 sound from the scene and generates audio signals, and an audip processor, which processes the 
audio signals responsive to a direction corresponding to a selected region in the scene. 

There is fijrther provided, in accordance with a preferred embodiment of the present 
invention, apparatus for producing a panoramic image, including: 

an image capture device, which generates video signals responsive to a panoramic 
25 scene; and 

a processor, which receives the video signals and applies a processing operation thereto, 
which varies as a fijnction of azimuthal angle, to produce a panoramic image of the scene. 

Preferably, the processing operation includes an image magnification, varying as a 
function of the azimuthal angle, wherein the processing operation varies the magnification to 
30 adjust for differences in the distances of objects in the scene from the image capture device. 

11 
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Alternatively or additionally, the processing operation includes an adjustment of the 
image brightness and/or image contrast and/or image color balance as a function of the 
azimuthal angle. 

There is also provided, in accordance with a preferred embodiment of the present 
5 invention, apparatus for image compression, including: 

at least one video sensor head, which captures an image of a scene; and 

a controller, which receives, digitizes and compresses the image, and which exchanges 

control signals with the at least one video sensor head so as to reduce the data size of the 

compressed image. 

10 Preferably, the controller provides timing signals to the at least one sensor head. 

Further preferably, the at least one sensor head includes a plurality of sensor heads, 
which receive respective partial images, and the controller controls the plurality of sensor heads 
and combines the partial images received therefrom to produce a panoramic image. 

Additionally or alternatively, the controller receives a video sensor parameter from the 
1 5 at least one sensor head and transmits the parameter to a remote site. 

In accordance with another preferred embodiment of the present invention, there is 
provided apparatus for displaying a panoramic image, including: 
a video display; and 

a processor, which receives panoramic image data, geometrically maps the data onto a 
20 generally annular surface, and renders at least a portion of the surface to the display. 

Preferably, the processor renders the generally annular surface in the form of two 
hemiannular strips, wherein the generally annular surface includes a generally cylindrical 
surface, and the processor renders the generally cylindrical surface to the display in a generally 
elliptical form. 

25 Preferably, the processor varies a point of view from which the surface is rendered. 

In a preferred embodiment of the invention, the video display, includes a generally 
annular display screen, including a back-projection screen and a video^{5rbjeGYb.r, which projects 
the image onto an inner side of the screen, so that the image is visibleyfrbm the outer side 
thereof 

30 There is moreover provided, in accordance with a preferred embodiment of the present 

invention, a method for displaying a panoramic image on a video display, including: 

defining an annular strip and a point of view outside a volume bounded by the strip; 

12 
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mapping the panoramic image onto the annular strip; and 

rendering an image of the strip, as seen from the point of view, to the video display. 
Preferably, mapping the panoramic image onto the annular strip includes: 
mapping a first segment of the image, in relative proximity to the point of view, onto a 
5 near hemiannulus; 

mapping a second segment of the image, relatively distant from the point of view, onto 
a far hemiannulus; and 

reversing at least a portion of the first segment. 

Preferably, defining the annular strip includes defining a generally cylindrical strip, and 
10 defining the point of view includes defining a point of view above an equatorial plane of the 
annular strip, so that the image of the strip rendered to the display is generally elliptical in 
shape. 

Preferably, the annular strip is rotated about an axis thereof, relative to the point of 
view, so as to shift the image rendered of the strip as seen from the point of view, substantially 
15 without engendering mechanical motion. 

There is also provided, in accordance with a preferred embodiment of the present 
invention, a method for displaying a panoramic image, including: 
providing a generally annular back-projection screen; 
mapping and rendering the panoramic image to the screen; 
20 projecting the image onto the screen from inside the annulus; and 

viewing the image from outside the annulus. 

In a preferred embodiment of the invention, the methods above include: 

receiving audio signals from an array of microphones placed adjacent to an area shown 

in the panoramic image; 
25 selecting a region of interest in the image; and 

processing the audio signals selectively, responsive to the "selected region. 

In accordance with another preferred embodiment of the present invention, there is 

provided a method for transmitting an image to a remote site, including: 
generating video signals using one or more video sensor heads; 
30 receiving and compressing the video signals for transmission; and 

exchanging control signals with the one or more video sensor heads so as to reduce the 

data size of the compressed signals. 

13 
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Preferably, exchanging the control signals includes receiving a sensor head control 
parameter and transmitting the parameter to the remote site, most preferably transmitting a 
change in a parameter relative to an earlier value thereof 

Funher preferably, generating the video signals includes generating signals using a 
5 plurality of video sensor heads, and exchanging the control signals includes selecting one or 
more of the plurality of sensor heads to transmit signals for use in updating a corresponding 
portion of a reference image, while other portions of the image corresponding to the non- 
selected sensor heads are substantially frozen. 

There is further provided, in accordance with a preferred embodiment of the present 
10 invention, a method for producing a panoramic image of a scene, including: 

positioning a plurality of substantially stationary video sensor heads to capture a 
respective plurality of panial images of the scene, each partial image having an angular extent 
substantially less than a total angular extent of the scene whose image is to be produced and 
having a region of overlap with a neighboring partial image; 
15 receiving video partial images from the plurality of sensor heads; and 

combining the partial images to produce the panoramic image. 

There is additionally provided, in accordance with a preferred embodiment of the 
present invention, a method for producing a panoramic image including: 

receiving video images covering a panoramic field of view; and 
20 processing the images variably as a function of azimuthal angle in the panoramic image. 

There is moreover provided, in accordance with a preferred embodiment of the present 
invention, a system for video teleconferencing, including at least one transmitting station, which 
comprises: 

a plurality of video sensor heads, positionable at or near the center of a table around 
25 which teleconferencing participants are seated so as to capture images of the participants on 
opposing sides of the table; and 

a transmission processor, which receives and combines video signals from the sensor 
heads to produce and transmit a panoramic image of the scene. 

Preferably, the plurality of sensor heads capture images of the participants along 
30 generally horizontal respective optical axes. 

Preferably, the system also includes at least one receiving station, which includes: 
a video monitor; 

14 
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a receiving processor, which receives the transmitted panoramic image and displays the 
image to the monitor; and 

user controls, which provide user commands to both the receiving processor and the 
transmission processor. 

Preferably, the user controls of the receiving station select one or more of the plurality 
of sensor heads of the transmitting station, and the video display is updated using images 
captured by the one or more selected sensor heads. 

In a preferred embodiment of the invention, the user controls of the receiving station 
include a menu generated by the transmitting station and displayed on the video display of the 
receiving station. 

In another preferred embodiment, the transmission processor exchanges sensor control 
parameters with the plurality of video sensor heads and transmits at least some of the 
parameters to the receiving processor, for use in rendering the image, wherein the transmission 
processor compresses the image for transmission, and wherein the receiving processor 
decompresses the image using the transmitted parameters. 

There is additionally provided, in accordance with a preferred embodiment of the 
present invention, apparatus for producing a stereoscopic panoramic image, including: 

a plurality of video sensor heads, which capture respective partial images of a scene; 

and 

a processor, which receives and combines the partial images from the sensor heads to 
produce a stereoscopic panoramic image of the scene. 

Preferably, the plurality of video sensor heads are substantially stationary. 

The present invention will be more fully understood from the following detailed 
description of the preferred embodiments thereof, taken together with the drawings in which: 
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BRIEF DESCRIPTION OF THE DRAWINGS 
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Fig. 1 is a schematic illustration of a teleconferencing facility, including panoramic 
image acquisition and display apparatus, in accordance with a preferred embodiment of the 
present invention; 

Fig. 2 is a block diagram illustrating signal flow between elements of the facility of Fig. 



Fig. 3 is a block diagram illustrating signal flow within the image acquisition and display 
apparatus of Fig. 1; 

Fig. 4A is a schematic, partly sectional illustration showing a side view of panoramic 
image acquisition apparatus, comprising a multifaceted mirror and a plurality of sensor heads, in 
accordance with a preferred embodiment of the present invention; 

Fig. 4B is a schematic top view of the sensor heads in the apparatus of Fig. 4 A; 

Fig. 5 is a schematic top view of the multifaceted mirror in the apparatus of Fig. 4 A; 

Fig. 6A is a schematic, pictorial illustration showing a detail of the multifaceted mirror 
shown in Fig. 5; 

Fig. 6B is a schematic illustration showing a head-on view of a single facet of the 
multifaceted mirror shown in Fig. 5; 

Fig. 6C is a schematic diagram illustrating optical beam paths associated with the detail 
of Fig. 6 A, 

Fig. 6D is a schematic diagram illustrating a detail of a multifaceted mirror, in 
accordance with an alternative embodiment of the present invention; 

Fig. 7A is a schematic illustration showing construction of a panoramic image, in 
accordance with a preferred embodiment of the present invention; 

Fig. 7B is a graph showing an intensity transfer function characteristic of panoramic 
image acquisition apparatus, in accordance with a preferred embodiment of the present 
invention; 

Fig. 8 is a schematic top view of a multifaceted mirror, in accordance with an alternative 
embodiment of the present invention; 

Fig. 9A is a block diagram which schematically illustrates the operation of a camera 
control unit, for use particularly in the panoramic image acquisition apparatus of Fig. 1, in 
accordance with a preferred embodiment of the present invention; 
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Fig. 9B is a block diagram which schematically illustrates details of the camera control 
unit of Fig. 9A; 

Fig. 10 A is a block diagram which schematically illustrates the operation of a panoramic 
image construction unit, in accordance with a preferred embodiment of the present invention; 
5 Figs. lOB and IOC are block diagrams which schematically illustrate details of the image 

construction unit of Fig. lOA; 

Fig. 1 1 is a schematic, partly pictorial illustration useful in understanding a method of 
panoramic image processing, in accordance with a preferred embodiment of the present 
invention; 

10 Fig. 12A is a flow chart illustrating a method of image compression, in accordance with 

a preferred embodiment of the present invention; 

Fig. 12B is a block diagram which schematically illustrates an image compression 
CODEC, in accordance with a preferred embodiment of the present invention; 

Fig. 13 is a schematic illustration showing a panoramic image display, in accordance 
1 5 with a preferred embodiment of the present invention; 

Fig. 14 is a schematic illustration showing a panoramic image display, in accordance 
with another preferred embodiment of the present invention; 

Fig. 15 is a schematic illustration showing a panoramic image display, in accordance 
with still another preferred embodiment of the present invention; 
20 Fig. 16 is a schematic illustration showing a panoramic image display, in accordance 

with yet another preferred embodiment of the present invention; 

Fig. 17 is a schematic, pictorial illustration showing panoramic image display apparatus, 
in accordance with a preferred embodiment of the present invention; 

Figs. 18A-18C are schematic illustrations, each showing two teleconferencing facilities 
25 interconnected by a communication line, in accordance with preferred embodiments of the 
present invention; and 

Fig. 19 is a schematic illustration of a stereoscopic panoramic imagev capture device, in 
accordance with a preferred embodiment of the present invention. '^- ^ 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Reference is now made to Fig. 1 , which schematically illustrates a teleconference facility 
30, based on apparatus and methods for image acquisition and display in accordance with a 
preferred embodiment of the present invention. Facility 30 is built around a panoramic imaging 
5 subsystem 40, including a panoramic image capture device 54, which is used to acquire a 
simultaneous, panoramic image of at least some, and preferably all, of panicipants 46 in the 
teleconference around a conference table 48. Device 54, which is preferably placed at or near 
the center of table 48, comprises a plurality of sensor heads, each of which acquires an image of 
a respective field of view, which images are then combined by subsystem 40 to form the 
1 0 panoramic image. The construction and operation of device 54 and subsystem 40 are described 
in detail hereinbelow. 

Facility 30 also includes a display monitor 50 and a plurality of local video displays 
(LVDs) 78, preferably comprising flat panel displays arranged in the center of table 48, as 
shown. The facility optionally also includes a conventional video camera, such as a pan-tilt- 

15 zoom (PTZ) controlled camera 32, as is known in the art, which is preferably adjacent to the 
monitor. A microphone array 52, for acquiring voice signals from the conference panicipants, 
is coupled to subsystem 40. Other elements of the facility optionally include a personal 
computer (PC) 34, a mouse 38 and/or other user interface device for controlling certain 
functions of subsystem 40, a presentation board 42 and an overhead camera 44 (also known in 

20 the art as a document camera). A CODEC 62 receives and, preferably, compresses video and 
audio signals generated by subsystem 40, and transmits the signals to a remote site, as well as 
receiving and distributing such signals from the remote site, preferably controlled by a remote 
control unit 36. Such elements of facility 30 are known in the art, but certain aspects of their 
construction and use in conjunction with subsystem 40 differ from conventional 

25 teleconferencing systems. 

Fig. 2 is a block diagram that schematically illustrates a video acquisition and display 
system 60, operative in the framework of facility 30. System 60 is built around acquisition 
subsystem 40, which is described below with reference to Fig. 3, along with CODEC 62. The 
CODEC may comprise any standard CODEC, known in the art, for example, a PictureTel 

30 Series 4000 CODEC, or otherwise as described in Chapter 12 of Video Demystified: A 
Handbook for the Digital Engineer, by Keith Jack (HighText Publications, San Diego). 
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Alternatively, CODEC 62 may comprise a special-purpose CODEC, designed to operate with 
subsystem 40, as described below with reference to Figs. 12A and 12B. 

The remote site may also have a panoramic imaging subsystem, like subsystem 40, or it 
may be based on imaging equipment known in the art. Preferably, subsystem 40 identifies 
5 whether the remote site has such a subsystem, by sending and receiving appropriate control 
signals via CODEC 62. The panoramic image capture and display capabilities of the subsystem, 
as described hereinbelow, may be realized regardless of whether the remote site has the 
subsystem, but certain aspects of these capabilities are enhanced when the remote site is 
suitably equipped. 

10 Fig. 3 is a block diagram showing details of panoramic imaging system 60 and 

subsystem 40. Subsystem 40 comprises a camera control unit (CCU) 74, which provides 
liming and control signals to image capture device (ICD) 54 and receives raw video output 
signals therefrom. CCU 74 preprocesses and digitizes these video signals and conveys them to 
an image construction unit (ICU) 64. The ICU combines the digitized video signals from the 

15 plurality of sensor heads in ICD 54 to form a single, substantially "seamless" digital panoramic 
image, as described in greater detail below. In the context of the present patent application, the 
expression "seamless" is meant to indicate that the panoramic image, which is constructed from 
a plurality of sub-images, as described below, appears to be substantially continuous and 
smooth over its entire extent, without spurious variations or artifacts at transitions between the 

20 sub-images. Preferably, ICU 64 also has RS-232 or other digital communications channels for 
communicating and exchanging control signals with CCU 74 and with CODEC 62, as well as 
with a touch screen 76 or mouse 38, for user control. 

A video input/output (VIO) unit 66 receives the digital panoramic image signals from 
ICU 64 and conveys the signals to CODEC 62. VIO 66 also receives audio and video signals 

25 via CODEC 62 from the remote site and distributes these signals respectively to speaker 68 and 
to local video displays (L\T3) 78, where they may be viewed by the conference participants, as 
well as to the display of touch screen 76. Pan/tilt/zoom (PTZ) signals may be received from the 
remote site, and are input to ICU 64 to control the acquisition and display of the panoramic 
image, as described below. VIO is preferably also coupled to receive and transmit VGA signals 

30 from PC 34 and video signals from overhead camera 44, which it conveys to ICU 64, and may 
include other I/O functions useful in teleconferencing, as are known in the art. 
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A voice activation unit (VAU) 70 receives audio signals from microphone array 52 and 
conveys these signals to VIO 66 for transmission via CODEC 62. The VAU is further coupled 
by a digital data and control link to ICU 64, so as to enable the following functions: 

• Voice-activated PTZ. The panoramic image produced by ICU 64 may be 
5 manipulated automatically so that the participant who is speaking is displayed in the 

center of the image (or in some other desired area of the image) or in an enlarged 
picture- in-picture (PIP) view, or both. 

• Audio beam-forming. The audio signals received from the multiple 
microphones in array 52 may be processed and combined, using signal processing 

10 techniques known in the art, such that the audio output heard at the remote site is that 

which comes from a selected direction within facility 30, for example, from the direction 
of a participant in the teleconference who is selected or centered in the panoramic 
image. The beam-forming may be carried out within acquisition and display system 60 
before transmission via CODEC 62. .Alternatively or additionally, some or all of the 

1 5 multiple audio signals received from the multiple microphones may be transmitted to the 

remote site, and the beam-forming can be performed there, preferably under the control 
of a user at the remote site. Still further alternatively or additionally, the beam-forming 
function may similarly be used to mute sound coming from a selected direction in 
facility 30, so that one or more of participants 46 can speak, for example, without being 

20 heard at the remote site. 

• Stereo sound. The remote site may be equipped with multiple speakers, and 
the audio signals from array 52 may be distributed to the speakers so as to give a 
stereophonic eflfect. 

Other elements of subsystem 40 preferably include user input and control interfaces, 
25 such as touch-screen 76, pointing device 38, preferably a wireless mouse, and a "Look-At-Me" 
button. The "Look-At-Me'' button (not shown in the figures) is a sort of acoustic pointer, 

which emits a special tone, preferably at a high frequency and/or short duration, substantially 
inaudible to humans, which is. recognized by VAU 70. These interface devices are designed so 
that subsystem 40 can interact with a similar subsystem at the remote site or, alternatively, so 
30 that it can be used opposite and interact with other teleconferencing video systems known in 
the art. An internal power supply 80 provides suitable electrical power to' all the elements of 
subsystem 40. 

20 
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PANORAiVnC nviAGE CAPTURE DEVICE 
Fig. 4A is a schematic, sectional, side view of panoramic image capture device (ICD) 
54, and Fig. 4B is a pictorial, sectional view of the device, as seen along line IVB-IVB in Fig. 
4A, in accordance with a preferred embodiment of the present invention. ICD 54 comprises a 

5 multifaceted mirror 100, having a plurality of planar facets 102, and a plurality of sensor heads 
92, preferably one for each facet of the mirror, commonly mounted on a frame 104. Each 
sensor head comprises a lens 96 and a detector array 98, preferably a CCD array, as is known 
in the art, contained in a housing 94. Although Fig, 4B shows ICD 54 comprising eight sensor 
heads 92, any other suitable number of sensor heads could be used. 

10 Each of the sensor heads receives an image from a respective angular field of view 1 10, 

represented in Fig. 4A by rays that converge to a respective apex 108. The effect of 
multifaceted mirror 100 and of the positioning of sensor heads 92 with respect thereto is such 
that apices 108 of all of the sensor heads are imaged to a common virtual apex 112 by 
respective facets 102. It will be appreciated, however, that the rays shown in Fig. 4 A represent 

15 only principal rays of sensor heads 92, and the actual ray traces for ICD 54 will be substantially 
more complicated. As is known in the art, rays that reach CCD 98 generally pass through an 
entrance pupil having a substantial lateral extent, as indicated schematically by a circle 106 in 
the figure. The entrance pupils of all of the sensor heads are imaged to substantially 
overlapping virtual pupils 1 15. 

20 ICD 54 is preferably covered by a protective case 105, including a transparent visor 

section 107 corresponding to fields of view 1 10 of sensor heads 92. As illustrated in Fig. 4 A, 
section 107 optionally comprises a refractive element, which functions in conjunction with 
lenses 96 to focus light onto CCDs 98. In this case, the visor is preferably formed so as to 
function as a cylindrical lens, thereby widening fields of view 1 10 in a vertical dimension. 

Z5 In addition to capturing a substantially "seamless" panoramic image, ICD 54 has a 

number of other advantages over teleconferencing cameras known in the art. It will be 
obscpv-ed in Fig. 4 A that the optical axes of fields of view 1 10 of sensor heads 92 are generally 
horizontal and are relatively high off table 48, near the eye level of participants 46. Because the 
axes are also near the top of ICD 54 and mirror 100, however, the ICD does not substantially 

30 prevent participants 46 from seeing one another across the table. Since ICD 54 has no moving 
parts or motors to make noise, microphone array 52 can be mounted on or in close proximity to 
the ICD, so that sound pickup from all sides of the table is unobstructed, and so that audio 

21 



wo 98/47291 - PCT/1L98/0014I 



signals can be easily correlated with the video images for the purposes of beam-forming, as 
described hereinabove. 

Fig. 5 is a schematic, bottom view of multifaceted mirror 100, as seen from the 
viewpoint of sensor heads 92. In this preferred embodiment, the mirror is seen to comprise 
5 eight facets 102, covering a fiill 360^, so that each of the sensor heads has, effectively, a 45° 
azimuthal field of view in the plane of Fig. 5. The outline of one of sensor heads 92 is shown 
schematically as a circle, superimposed on one facet 102a. It will be appreciated, however, that 
in other embodiments of the present invention, not shown in the figures, the multifaceted mirror 
may comprise more or fewer facets, and ICD 54 may be set up to cover less than the full 360^. 
10 In still other preferred embodiments, as described below, for example, with reference to Fig. 
13, there may be more than one sensor head per facet of the multifaceted mirror. 

Fig. 6 A is a schematic, pictorial representation of a portion of mirror 100, including two 
neighboring facets 102a and 102b, and Fig. 6B is a schematic, head-on view of one of facets 
102. As seen most clearly in Fig. 6A, facets 102a and 1 02b are separated by a generally 
\> triangular baffle 120. The purpose of this bafile is to allow rays to reach CCD 98 in each of 
sensor heads 92 only from the sensor head's respective facet 102, and to prevent rays reflected 
from neighboring facets from reaching the CCD. Thus, ray BA, shown in Fig. 5, is allowed to 
reflect from facet 1 02a and enter the respective sensor head, but ray C A, reflected from facet 
102b, is effectively prevented from entering the sensor head by baffle 120. Failure to block rays 
20 such as ray CA will result in "ghost images" being received by CCD 98, with a consequent 
deleterious effect on the quality of the panoramic images produced by ICD 54. Panoramic 
cameras based on pyramidal mirrors known in the art do not address this difficulty. 

Fig. 6C is a schematic optical diagram, in a sectional plane through mirror 100 taken 
along line VIC-VTC in Fig. 6A, useful in understanding the fiinction of baffle 120. A principle 
25 ray 114 reflects off facet 102a and reaches apex 108. A spurious ray 116 reflects off facet 
102b, parallel to ray 1 14, and would pass through pupil 106 into the sensor head corresponding 
to facet 102a, if ray 116 were not blocked by baffle 120. The baffle must extend out far enough 
from the joint between the facets to block all such spurious rays. ..-J^' 

Fig. 6D is a schematic diagram, from the same point of view as Fig. 6C, showing a 
30 detail of mirror 100 in accordance with an alternative embodiment of the present invention. In 
this case, mirror 100 includes baffles 118, at the center of each of facets 102, in addirion to 
baffle 120 at the joint between the facets. The presence of baffle 118 on facet 102b, for 
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example, blocks spurious rays that otherwise would pass outside the outer end of baffle 120 
and reach the entrance pupil of the sensor head associated with facet 102a. Consequently, 
baffle 120 can be made shorter, so that mirror 100 is more compact and less prone to damage. 
Alternatively, baffles 118 may be used alone, without baffles 120, and/or additional baffles may 

5 be placed at other suitable locations along facets 102. 

rig. 7 A is a schematic representation of partial images 121 and 122 of a teleconference 
participant 123, as they are received by two of sensor heads 92 viewing the participant by way 
of respective, neighboring facets 102b and 102a of mirror 100. Preferably, lenses 96 are so 
designed and focused, and sensor heads 92 are so positioned relative to respective facets 102, 

10 that fields of view 110 of neighboring sensor heads substantially overlap in the azimuthal 
dimension, creating an overlap region 124. Partial images 121 and 122 are shown as having 
trapezoidal forms in the figure to reflect keystone distortion of the images, which results from 
the structure of multifaceted mirror 100, as is known in the art. 



15 images of Fig. 7 A. The intensity transfer is shown as a function of azimuthal angle along an 
arbitrary horizontal line 127 passing through images 121 and 122, preferably a line 
corresponding to a common row of pixels in detectors 98 of the respective sensor heads 92. 
Whereas the intensity transfer is substantially constant, at a normalized value of 1, over most of 
the area of the partial images, in overiap region 124 there is vignetting, due inter alia to the 

20 finite aperture afforded by facets 120, which reduces the image intensity. Preferably, although 
not necessarily, mirror 100 is so designed that the sum of the intensity transfer functions for the 
two partial images in the overiap region, shown in the figure by a dashed line 125, is 
approximately equal to 1, as well. For example, referring to Fig. 6B, the edges of facet 102 
adjacent to baffles 120 may be tinted to reduce their reflectivity. 

25 Referring again to Fig. 7A, a method for constructing an effectively cylindrical 

panoramic image 126 will be described, in accordance with a preferred embodiment of the 
present invention. For each point along line 127, ICU 64 (Fig. 3) receives digitized image data 
from ecu 74. For the most part, only a single data value is received for each point, but in 
overlap region 124, two data values are received, from neighboring sensor heads 92, and are 

30 added together. If the sum of the intensity transfer functions, as described above, deviates 
significantly from I, a multiplicative correction factor is preferably applied to the data values, so 
that the intensity of the image will vary smoothly over the overiap region. This process is 



Fig. 7B is a graph representing intensity transfer fijnctions corresponding to the partial 
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continued over the entire azimuthal extent of line 127, preferably over 360^, and is carried out 
for all other horizontal lines in the image in succession, preferably corresponding to successive 
rows of pixels in detectors 98. 

In conjunction with sununing partial images 121 and 122 so that panoramic image 126 
has a smooth transition in overiap region 124, the partial images are preferably mapped, either 
before or after summing, to a generally cylindrical image surface, indicated by the curved lines 
corresponding to image 126 in Fig. 7A. Panoramic image 126 is thus approximately equivalent 
to an image of the scene that would be captured by a camera with a cylindrical detector array (if 
such a detector array and camera could be produced). The mapping may be achieved, at least 
in part, by producing lenses 96 with such barrel distortion as to offset the keystone distortion 
mentioned above. Alternatively or additionally, the rows of partial images 121 and 122 or of 
combined image 126 may be mapped onto the cylindrical surface using any suitable method of 
interpolation known in the an. 

Fig. 8 is a schematic top view of a multifaceted mirror 130 having curved facets 132, in 
accordance with another preferred embodiment of the present invention. Facets 132 are 
preferably convex, either spherical or aspheric, for example, comprising surfaces of revolution 
of conic sections or toroidal surfaces. As illustrated in Fig. 8, the facets typically have radii of 
curvature that are greater than the radius of a circle circumscribing mirror 130. The curved 
surfaces of facets 132 produce partial images in sensor heads 92 having reduced keystone 
distortion and more closely approximating a cylindrical image surface than partial images 
captured using planar mirror facets. The refractive power of the curved facets (in addition to 
that of lenses 96) also enables mirror 130 to be made smaller for a given sensor head geometry 
than mirror 100 could. 

In any case, even with planar facets 102, in preferred embodiments of the present 
invention, the deviation of partial images 121 and 122 from the desired cylindrical surface of 
image 126 is generally small and is concentrated at the upper and lower edges of the images. 
Such deviation will, ftirthermore, be masked at least in part by the summation of the images in 
overlap region 124. Therefore, the inventors have found that even zero-order interpolation is 
sufficient to produce image 126 with adequate image quality and without obvious distortions. 
The method of the present invention, by virtue of using overlapping neighboring images and 
cylindrical image mapping, as described above, allows panoramic image 126 to be constructed 
rapidly and at low computational cost, relative to methods known in the art for correcting and 
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combining contiguous partial images, as described in the above-mentioned U.S. patents 
5,187,571 and 5,444,478. for example. 

Additionally, in a preferred embodiment of the present invention, mapping of partial 
images 121 and 122 is further adjusted to compensate for the varying distances of the 

5 participants in the teleconference from ICD 54. As shown in Fig. 1, seats 46 are positioned at 
different distances from ICD 54 at the center of table 48, so that without image correction, the 
participants will appear in panoramic image 126 to be of different sizes. Therefore, a mapping 
function having a variable magnification as a function of azimuthal angle is preferably applied to 
partial images 121 and 122, so that the figures of all the participants are of approximately equal 

1 0 size in the panoramic image. 

As noted above with reference to Fig. 3, ICU 64 and CCU 74 preferably perform 
additional camera control and image processing functions, so as to enhance the quality of 
panoramic image 126 and ensure a smooth transition and an absence of 'seams" between the 
parts of image 126 corresponding to different panial images 121 and 122. CCU 74 provides 

15 timing signals to sensor heads 92, including mutually-synchronized pixel clocks, horizontal and 
vertical blanking signals and electronic shutter signals. Digitization of the video outputs of the 
sensor heads is preferably also synchronized to the pixel clocks. 

Fig. 9A is a block diagram showing details of CCU 74, in accordance with a preferred 
embodiment of the present invention. CCU 74 comprises a plurality of input stages 133, 

20 preferably one stage for each of sensor heads 92 in ICD 54. A clock generator 134 and a 
synchronization signal generator 135 generate a common pixel clock and horizontal and vertical 
blanking signals for sensor heads 92 and for input stages 133. A CPU 136, which may 
comprise any suitable microcontroller known in the art, controls the operation of the input 
stages, for example, varj'ing the gain and offset settings for each of the sensor heads, and, 

25 optionally, communicates with ICU 64 and with other elements of subsystem 40. 

Fig. 9B is a block diagram showing details of one of input stages 133. Each of input 
stages 133 comprises a correlated-double-sampler (CDS) 137, as is known in the art, which 
receives a raw video signal from the respective sensor head 92 and provides a sampled-and-held 
output to an A/D converter 138. The digital output of converter 138 is preferably corrected, 

30 pixel by pixel, by multiplication in a multiplier 141 with a correction factor, which is stored in a 
memory 139. The corrected, digitized signal is then processed by a video processor 143, 
preferably to generate a 24-bit digital RGB video signal. All the elements of input stage 133 are 
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commonly timed by inputs from clock generator 134 and sync generator 135. Gain, offset and 
correction factor data are preferably received from CPU 136, as described above. 

Video processor 143 preferably comprises an iSP 2000 processor chip, manufactured by 
i Sight Ltd., of Tirat Hacarmel, Israel. Alternatively, other video processors known in the art 
5 may be used. Furthermore, although in the embodiment shown in Fig. 9B, the raw video 
signals are first digitized and then processed, other video processing configurations known in 
the art may also be used. For example, the video signals may be processed using analog 
components and methods, and only afterwards digitized, and/or the output of CCU 74 may 
comprise a Y/C or other type of video signal. 

10 Fig. 1 OA is a block diagram which schematically illustrates the functions of ICU 64, in 

accordance with a preferred embodiment of the present invention. ICU 64 receives multiple 
digital video inputs from CCU 74, preferably eight such inputs (labeled CAMO through 
CAM7), each of the inputs 24 bits wide, corresponding to the eight sensor heads 92 shown in 
Fig. 4B. These inputs are combined by a panoramic image composer 140 to form the 

15 panoramic image, and a close-up image composer 142 to form a close-up image of a region of 
interest (ROI), as described further hereinbelow. ICU 64 also includes an external video input 
146, which receives video signals from VIO 66, for example, captured by camera 44, as well as 
timing control and synchronization signals. A background color unit 148 is used to fill in areas 
of the picture that are not occupied by video images. The panoramic and close-up images 

20 formed by composers 140 and 142, respectively, together with the external video input and 
background color are combined by a multiplexer (MUX) 144 and are output via an overiay 
generator 152 to VIO 66. A source selector 150 determines, for each pixel of the output 
image, which of the inputs to MUX 144 is to be selected. 

Fig. 1 OB is a block diagram showing details of panoramic image composer 140, in 

25 accordance with a preferred embodiment of the present invention. Each field of digital video 
data received from each of the sensor head inputs from CCU 74 is stored serially in a respective 
memory 160. The data are then read out in proper order to a FIFO memory 162, under the 
control of read/write logic 161, so as to form a single, substantially continuous panoramic strip. 
Preferably, only one of the two video fields in each frame that is output by sensor heads 92 is 

30 stored in memory 160, for example, the odd field. In this case, the data can then be read out if 
memory 160 during the even field period, without the need for double buffering. The reduced 
vertical resolution due to operating in such a field mode, rather than full frame, is in accordance 
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with well-known standards of video teleconferencing, such as the H.261 Common Image 
Format. 

Logic 161 includes a RAM select FIFO 172, which indicates which of memories 160 is 
to be read out, and read FIFO's 168 and 170, which generate high and low address bytes from 

5 reading from the memories. A strip rotation FIFO 1 64 controls a viewing angle from which the 
panoramic image is to be rendered, as described in greater detail below. PAL's 166 and 174 
receive data from FIFO's 164 and 172, respectively, and generate the control inputs to memory 
160 that determine the order of memory readout to produce the desired viewing angle. 

If desired, a portion of the panoramic image generated by composer 140 may be 

10 suppressed, preferably by interrupting the normal writing of camera data to an appropriate 
address range of memories 160. The suppressed portion of the image may be dimmed, blacked 
out or frozen. Such a function may be invoked, for example, if one or more of participants 46 
do not wish to be seen in the image transmitted to the remote site. In this case, the dinmied or 
frozen portion of the image, or a blank image, is read out of the address range into FIFO 162 in 

1 5 place of a live image. Alternatively, a video loop corresponding to the portion of the image to 
be blacked out may be stored and then replayed into memories 160, in order to give the 
appearance of a live image. 

Fig. IOC is a block diagram showing details of close-up composer 142, in accordance 
with a preferred embodiment of the present invention. The sensor head inputs from CCU 74 

20 are multiplexed by multiplexers 180 in odd and even groups, so that the digital data from any 
one of sensor heads 92 is channeled separately from that of its immediate neighbors. A 
programmable delay 182 is applied to the data from each of the sensor heads, based on 
preprogrammed instructions stored in a ROM 184. The delay, which is preferably controlled to 
an accuracy of 1/4 pixel, provides linear interpolation of the input partial images to correct for 

25 image distonion. The data are then stored by FIFO's 186. Readout from the FIFO's is 
selected according to the view angle and boundaries of the close-up image to be generated, 
based on preprogrammed and/or user-controlled inputs. • ^ 

The partial images, output from FIFO's; 1.86, are merged* by a programmable 
multiplier/adder 188, which blends neighboring partial images at their edges as illustrated in 

30 Figs, 7 A and 7B. The blending is performed according to an interpolation function given 
generally by: 

27 



wo 98/47291 



PCT/1L98/00141 



Y = aXi +(l-a)- ^2 (0 

wherein Y represents an output pixel value, and X2 are the input pixel values, and a is a 
parameter input for each pixel from a ROM 190. Preferably, a varies between 0 and 1 over a 

5 range about 64 pixels wide, representing an area of overlap between two adjacent partial 
images 121 and 122, out of a typical total width of 768 pixels in each partial image. 
Alternatively, a may have a value greater than or equal to one for some or all of the pixels over 
the overlap area when the intensities of the panial images are attenuated optically by mirror 
facets 102, as described above. 

10 Although in the embodiment of Figs. 1 OA- IOC, only close-up composer 142 includes 

the image blending function of multiplier/adder 188, it will be appreciated that a similar circuit 
could be used to blend the edges of the partial images in the panoramic image produced by 
panoramic composer 140. Due to the lower resolution of the panoramic image, however, as 
well as the optical blending features of mirror 100, as described above, such image blending is 

1 5 generally not required in the panoramic composer. 

The merged image from block 188 is input to deinterlacing FIFO's 192 and from there 
to double-buffered line stack mirror FIFO's 194. FIFO's 192 convert each two fields of 
interlaced image data into a single, non-interlaced frame for subsequent processing. FIFO's 
194 reverse each line of video, to compensate for the left-right reversal introduced by mirror 

20 100. A zoom unit 196, preferably based on a GM833x3, produced by Genesis Microchip ltd. 
of N4arkham, Ontario, controls the readout from FIFO's 192 and 194 and performs 
interpolation operations on the non-interlaced data to provide digital zoom in the close-up 
image. The output of unit 196 is received by a FIFO 198, which re-interlaces the image and 
outputs it to MUX 144. 

25 Although ICU 64 as shown in Figs. 1 OA- IOC and described above produces both 

panoramic and close-up images, and combines them together in a single video output image 
with external video and background color, other preferred embodiments of the present 
invention need not include all of these elements. For example, close-up composer 142 may be 
eliminated, and ICU 64 can be made to transmit only a panoramic image. If a remote site 

30 receiving the panoramic image is suitably equipped, a close-up image can be produced at the 
remote site by cropping a region of interest from the panoramic image and digitally zooming in 
on the region, as is known in the art. Alternatively, the partial images output by CCU 74 may 
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be transmitted to the remote site, and the panoramic and close-up images may then be 
composed by a unit similar to ICU 64 at the remote site. Those skilled in the art will appreciate 
other configuraiiona! variations that may be implemented, based on the principles of the present 
invention. 

5 Fig. 11 is a schematic illustration useful in understanding the functions of video 

processing circuitry in CCU 74, in accordance with a preferred embodiment of the present 
invention. CCU 74 adjusts and controls the brightness, contrast, and other aspects of the video 
signals in the partial images. Such adjustment is preferably carried out before digitization of the 
signals, for example, by adjusting the gain of preamplifiers that amplify the video signals before 

10 digitization. 

In the example shown in Fig. 11, the lighting in teleconferencing facility 30 is not 
uniform, including brighter areas near light sources 226 and dimmer areas 227. Signal 
processing circuitry 228 in CCU 74 determines a brightness function /, dependent on azimuthal 
angle (|), illustrated by a graph 225 shown in the figure. Circuitr>' 228 then sets the gain and/or 

1 5 other aspects of the video signal processing to vary as a function of azimuthal angle, normally 
with a higher gain in more dimly-lit areas of the image, i.e., generally inversely responsive to 
y((t)). Rather than setting the gain (or other parameters) to a single fixed value for all of sensor 
heads 92, or to a fixed, individual value for each of the sensor heads, the gain is set to vary 
substantially smoothly, so that the brightness of the participants appears to be generally uniform 

20 over all of the panoramic image 126 Other image parameters, such as contrast, may similarly 
be controlled as a fijnction of azimuth. 

These functions of CCU 74 are preferably controlled by ICU 64, responsive to 
characteristics of the image and to changes therein. Additionally or alternatively, ICU 64 
preferably performs image processing and enhancement functions so as to enhance the quality 

25 of the panoramic image. In a preferred embodiment of the present invention, ICU 64 controls 
CCU 74 and sensor heads 92 and processes image data received therefrom to enhance the 
dynamic range of the panoramic image, using methods of Adaptive Sensitivity, as described in 
U.S. Patents 5,144,442 and 5,247,366, which are incorporated herein by reference. In another 
preferred embodiment, ICU 64 processes the image to increase the apparent image resolution, 

30 as described, for example, in an article by I. Vitsnudel and Y.Y. Zeevi, entitled ''Neural 
Network- Aided Design for Image Processing," SPIE Proc. (1991), which is incorporated 
herein by reference, or as is otherwise known in the art. 
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It will be clear to those skilled in the art that the video processing functions of CCU and 
ICU described above may be carried out in a wide variety of ways, using readily-available 
analog and digital electronic components and signal processing methods. Furthermore, although 
in the preferred embodiments described above, some of the image processing, enhancement and 
5 control ftmctions have been associated with specific units within panoramic imaging subsystem 
40, it will be appreciated that these functions could similarly be performed by other units in the 
subsystem. 

Fig. 12A is a flow chart showing a method for content-responsive compression of video 
images for transmission by CODEC 62, in accordance with a preferred embodiment of the 

10 present invention. Such compression is preferably compliant with video compression standards 
known in the art, such as JPEG and MPEG compression. Preferably, ICU 64 comprises a 
processor which compresses the digital video signals conveyed fi-om ICU 64 to VIO 66, for 
transmission via the CODEC, which may then comprise a standard CODEC, known in the art. 
Alternatively, the CODEC may comprise a special-purpose device, adapted to work with 

15 subsystem 40, whereby the compression is performed by the CODEC, with appropriate 
feedback to subsystem 40. 

Preferably, the entire panoramic image 126 is transmitted periodically, in compressed 
form, as a base or reference image. To begin the compression process, panoramic image 126 is 
constructed, as described above. In order to optimize the efficiency of compression, i.e., to 

20 reduce the volume of compressed data that must be transmitted, ICU 64 (or CODEC 62) 
preferably identifies areas within the image that include visual information of importance and 
other areas that contain only static image background. The ICU then orders CCU 74 to adjust 
timing signals that it provides to ICD 54 and video processing parameters applied to the signals 
therefrom, so as to optimize the signals for compression. The base image is then compressed 

15 and transmitted. 

For subsequent video frames, only certain portions of the image are transmitted and are 
used to update the panoramic image displayed at the remote site. Specifically, ICU 64 
preferably acquires and updates only a region of interest (ROI) within the panoramic image. 
Preferably, the ROI comprises one or two partial images, indicated by VAU 70 as including the 
"^^J participant who is speaking. Alternatively, the ROI may be selected by a user at the remote 
site. Furthermore, within the ROI, preferably only the figure of the participant of interest is 
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updated, while areas of background are identified, using image analysis methods known in the 
art, and are left unchanged. 

From time to time, for example, once per second, the entire panoramic base image is 
updated by ICU 64 and is compressed and retransmitted, as described above. In between these 
5 updates, the ICU continues to transmit ROI image data. If the ROI changes, as indicated by 
VAU 70 or by user input, the new ROI is identified and becomes the basis for the frame-to- 
frame image updates. 

Fig. 12B is a block diagram which schematically illustrates image content-responsive 
compression functions performed by CODEC 62, in accordance with a preferred embodiment 

10 of the present invention. Each time a new image is received from VIO 66, the image is 
compared with an updated preceding image, stored in memory 212, by a motion estimator 214. 
The estimated motion is used in controlling subsequent image compression steps. The 
preceding image is used to produce an image background fill 220, which is subtracted from the 
new image by an image subtractor 216, dependent on the estimated image motion. An image 

15 compressor 218 then compresses the subtracted image for transmission over the 
communication line. 

The compressed output preferably carries information relating to image motion, 
typically motion vectors, and relating to camera parameters received from CCU 74, for use in 
reconstructing the image at the remote viewing site. Such camera parameters preferably 

"20 include gain factors, electronic shutter speeds and color balance parameters, which are applied 
by CCU 74 in producing the video images. When a variation in image conditions causes any of 
the camera parameters to change, it is enough for CODEC 62 to transmit data corresponding to 
the change in the camera parameters. As long as a remote site receiving the data is similarly 
equipped with a CODEC capable of receiving and interpreting the camera parameter 

25 information, the image can be updated at the remote site based on the parameters, and without 
the need for transmitting the entire image as would by required by systems known in the art. 

For each new image frame, a decompressor 222 reconstructs the complete preceding 
image based on the transmitted, compressed image. This preceding image is input to memorj' 
212, as described above, and to a. background extractor 223, which identifies the areas of the 

30 image that comprise only static background. The identified areas are processed by a 
background adapter 224, responsive to the camera parameters, and the resultant background 
image is input to background fill 220, for subtraction from the new image as described above. 
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The compressed output of compressor 218 need not include any information regarding 
unchanged background areas of the image. As a result, substantially more efficient image 
compression is achieved, while the transmission of camera and motion parameters can be used 
to enhance image reconstruction at the remote site. 

5 In a further preferred embodiment of the present invention, not shown in the figures, 

CODEC 62 also produces timing and control signals that are input to CCU 74 and/or to other 
elements of subsystem 40. For example, in order to reduce transmission bandwidth, the 
CODEC may command the CCU to convey video signals firom only a single one or a few of 
sensor heads 92, while partial images produced by the other sensor heads are effectively firozen. 

10 However, when the CODEC is preparing to transmit an update of the full panoramic image 
frame, it will command the CCU to unfreeze and convey for transmission images from all of the 
sensor heads. 

PANORAMIC IMAGE DISPLAY AND CONTROL 
Fig. 13 is a schematic illustration showing a panoramic teleconference image 231, as 

1 5 displayed on monitor 50, in accordance with a preferred embodiment of the present invention. 
Preferably, image 23 1 is acquired using an image acquisition subsystem at the remote site that is 
similar to subsystem 40, so that image 231 has the characteristics of panoramic image 226. 
Alternatively, however, the methods of displaying panoramic images represented by Fig. 13 and 
subsequent figures and described hereinbelow may be applied to other types of panoramic 

20 images and methods of panoramic, image acquisition known in the art. Alternative apparatus 
and methods for displaying the panoramic image are described below with reference to Fig. 17. 

As described above, panoramic image 231 is preferably mapped onto a substantially 
cylindrical image surface. This image surface is rendered to monitor 50 in the form of an 
annular strip 230, created by viewing the image surface from a virtual point of view outside the 

25 volume contained within the cylindrical surface and, preferably, substantially above its 
equatorial plane. As a result, both a front hemiannulus 239 and a rear heniiannulus 237 of strip 
230 are visible in the rendered image. As shown in Fig. 13, when seeh.fr,om the virtual view 
point, strip 230 assumes a generally elliptical form. Rendering of the panoramic image to 
annular strip 230 is preferably performed by an image construction unit such as ICU 64 at the 

30 remote site. Alternatively the image may be so rendered by ICU 64 at facility 30, and 
transmitted by CODEC 62 in this form. 
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Annular strip 230 has an inner surface 235 and an outer surface 233. Since image 231 
is acquired from a point of view at the center of the strip, preferably as described above, the 
image before rendition is mapped only onto inner surface 235. In order for the image to be 
visible when rendered onto front hemiannulus 239, the inner surface of the hemiannulus is first 
mapped onto outer surface 233 thereof Preferably, figures on hemiannulus 239, such as 
figures of participants 232 and 234, and other direction-sensitive image elements, such as 
writing 241, are automatically identified and are each left -right reversed in the course of 
mapping surface 235 to surface 233, using image processing methods known in the art. 
Otherwise, these figures and other elements would appear backwards to a viewer of monitor 
50. Figures 236 on hemiannulus 237 are shown without such reversal. 

Although Fig. 13 shows a generally cylindrical image surface mapped to strip 230, it 
will be appreciated that the principles described above may be used to map and render images 
having other geometries. For example, image 231 may be mapped onto an annulus having an 
oblong or otherwise irregular form, corresponding to differing distances of participant figures 
232, 234 and 236 from the center of the conference table. The cylindrical form of strip 230, 
taken together with the method of generating cylindrical panoramic image 226, has an 
advantage over other geometries, however, in that the strip can be rotated by any desired angle, 
as described below, without the necessity of geometrically remapping the image before it is 
rendered. As shown in the figure, all of the participants are shown on strip 230 at 
approximately the same size. They will also preferably be displayed with generally the same 
levels of brightness, color balance and other image parameters, based on the methods of angle- 
dependent image processing described above. 

Another major advantage of the method of display illustrated by Fig. 13, by comparison 
with panoramic displays known in the an, is that all of the participants are shown together in 
the image in positions that are intuitively representative of their actual positions at the remote 
teleconference site. It will be appreciated that this type of panoramic display does not require a 
special, wide-aspect monitor screen, as is frequently used in displaying panoramic images, and 
is equally suited for different monitor sizes and types. 

Preferably, a user viewing monitor 50 can control various aspects of the rendition, for 
example, the virtual point of view from which image 231 is rendered. Most preferably, the user 
can rotate strip 230 about its central axis, so as to alter which one of participant figures 232, 
234 and 236 is seen at the center of the monitor. Such rotation is equivalent to panning a 
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moving camera placed at the center of table 48. Unlike systems known in the art, however, the 
principles of the present invention allow the rotation to be accomplished substantially without 
engendering any mechanical motion. Image rotation is accomplished by giving a suitable 
"rotate" command, or, preferably, by selecting a region to be centered, whereupon strip 230 is 

5 rotated to the appropriate orientation. Additionally or alternatively, strip 230 may rotate 
automatically, responsive to which of the participants at the remote site is speaking, as 
indicated by VAU 70 (Fig. 3). If image 231 has been acquired and constructed at the remote 
site so as to have a substantially cylindrical form to begin with, like image 226, described 
above, then the image rotation can be accomplished without substantial geometrical remapping. 

10 Fig. 14 schematically illustrates an alternative preferred embodiment of the present 

invention, wherein monitor 50 displays both panoramic image 23 1 and an enlarged picture-in- 
picture (PEP) image 240 of participant 232. PIP image 240 provides a close-up view of a 
region of interest, for example, the region that includes the participant making a presentation in 
tliis case. Additional PIP images of other participants, or of other images of interest, such as of 

15 documents viewed by overhead camera 44 or a graphic display generated by PC 34 (both 
shown in Fig. 1), may also be superimposed on panoramic image 231 if desired. The region to 
be shown in PIP image 240 may be selected as described above, i.e., by user selection or 
responsive to VAU 70. Preferably, the selected region is marked by a border 242 and/or by a 
cursor 244, and/or has its brightness increased relative to other areas of the image. 

20 Alternatively or additionally, the user may "cut and paste" a portion of panoramic image 23 1 to 
form PBP image 240 and may further control the magnification, i.e., the zoom, and the tilt angle 
(elevation) at which the PIP image is rendered. 

Monitor 50 may also include additional PIP images, including, for example, an image 
246 of a document captured by camera 44, or an image of another participant, such as the 

25 preceding speaker PEP images 240 and/or 246 may display local participants, too, in addition 
to or instead of participants at the remote site. Furthermore, system 60 may be programmed so 
that after a given participant has been speaking for a predetermined period of time, between 10 
and 30 sec, for example, image 240 or 246 switches to another participant, such as the 
preceding speaker. Alternatively or additionally, the system may be programmed under certain 

30 conditions to automatically change its display mode and/or switch to display other participants. 

Fig. 15 schematically illustrates still another preferred embodiment of the present 
invenrion, wherein monitor 50 displays panoramic image 231 with a portion 248 of the image 
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enlarged corresponding to the region of interest. Selection of the ROI and rotation of strip 230 
are preferably performed substantially as described above. In the embodiment of Fig. 15, 
however, portion 248 of strip 230 is intentionally distorted, so that the ROI is shown in its 
proper place within the strip, rather than in a PEP window. 

5 Fig. 16 is a schematic illustration of monitor 50 showing an alternative method of 

displaying panoramic image 231, in accordance with another embodiment of the present 
invention. In this case, the generally cylindrical image surface is "unrolled," and is rendered in 
the form of a long strip 250, showing teleconference participants 252 and 254. A selected 
participant 252 is shown at the center of strip 250 and is also displayed in a PDP window 256. 

10 In other respects, this embodiment is similar to that shown in Fig. 14 and described with 
reference thereto. 

Fig. 17 is a schematic, pictorial illustration showing a generally circular panoramic 
image display 257, in accordance with a preferred embodiment of the present invention. 
Display 257 is particularly suited for displaying images in the form of annular strip 230, 

15 constructed as described above, but it may also be adapted for displaying other types of 
panoramic images. In facility 30, shown in Fig. 1, display 257 would preferably be placed at 
the center of table 48, adjacent ICD 54, where the display could conveniently be viewed by all 
the participants. Although display 257 is shown in Fig. 17 as having a spherical, "crystal ball" 
shape, it will be understood that other shapes could equally be used, for example, a cylindrical 

20 shape. 

Display 257 comprises a video projector 258, as is known in the art, which receives 
video signals from VIO 66 corresponding to strip 230, suitably geometrically transformed and 
rendered, and projects a corresponding video image upwards toward facets 259 of a 
multifaceted mirror 255. Facets 259 reflect the image toward an annular back-projection 

25 screen 261, whereupon the image of strip 230 is seen on the outside of the screen and extends 
all around the outside of display 257. Preferably, facets 259 are only partially reflecting, so that 
a viewer on one side of display 257 will see both sides of the panoramic image, by viewing the 
far side of the image through the partially-reflecting facets. It will be appreciated that many of 
the image control functions described above, such as rotation of the panoramic image 

30 responsive to selection of a region of interest, can similariy be performed using display 257. 

Figs. ISA and 18B are schematic illustrations showing two different teleconferencing 
configurations in which image capture subsystem 40 and the above-described methods of image 
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display may be used, in accordance with preferred embodiments of the present invention. In 
Fig. ISA, two teleconferencing facilities 30a and 30b, both including a respective subsystem 
40a and 40b, like subsystem 40, and similar in other respects to facility 30, shown in Fig. 1 and 
described above, are connected via a communications link 260, which may be of any suitable 
type known in the art. In Fig. I SB, on the other hand, facility 30 is connected via link 260 to a 
conventional teleconference facility 262, including a controller 264, a monitor 266 and a 
camera 268, preferably with pan-tilt-zoom (PTZ) control, as is known in the art. Facility 262 
could be equipped, for example, with a video conferencing system produced by PictureTel 
Corporation, mentioned above, such as the Concorde 4500, Venue 2000 or System 4000ZX. 
Although there are certain aspects of the present invention that are realized only in the 
configuration of Fig. ISA, both configurations allow panoramic images, such as image 231, as 
described above, to be displayed and controlled at both of the teleconferencing facilities that are 
involved. 

In the configuration of Fig. ISA, subsystems 40a and 40b include respective panoramic 
image capture devices (ICD) 54a and 54b and are coupled to respective monitors 50a and 50b. 
Thus, at both ends of the teleconference, panoramic images are captured, as described above, 
and are conveyed over link 260. Preferably, images captured by ICD 54a are conveyed to 
subsystem 40b in the form of individual partial images, such as images 121 and 122 shown in 
Fig. 7A, as they are captured by individual sensor heads 92 (Fig. 4A) of the ICD. The partial 
images are processed and assembled by system 40b into a panoramic image, which is displayed 
on monitor 50b, preferably as shown in Fig. 13, 14, 15 or 16. Participants in facility 30b can 
directly control various aspects of the panoramic image display on monitor 50b, using image 
processing functions provided by subsystem 40b. 

Preferably, as described above, not all of the partial images are transmitted over link 
260 for each video fi'ame. Rather, the sensor head or sensor heads whose partial images are to 
be transmitted are determined responsive to selection of a ROI by' a user, either at the sending 
end (facility 30a) or the receiving end (facility 30b), or are selected automatically, based on the 
direction of audio signals received by subsystem 40a, for example. Further preferably, only 
frame-to-frame changes in the selected partial images are transmitted. 

The configuration of Fig. 18B also allows panoramic images, as shown in Figs. 13-16, 
to be viewed at both ends of the teleconference, although in this case the image processing 
fiinctions are substantially all performed by subsystem 40. To acquire and display a panoramic 
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15 



20 



25 



image of the panicipants in facility 262 on monitor 50, subsystem 40 sends a command to 
controller 264, instructing camera 268 to pan, as indicated by an arrow 270. Subsystem 40 
receives multiple images captured by camera 268 at various pan angles, and assembles the 
images into a single panoramic image, such as image 23 1 . Subsequently, camera 268 is aimed 
automatically or under user control, preferably to capture images of whichever participant or 
participants in facility 262 are speaking. Subsystem 40 receives these images and uses them to 
update the panoramic image displayed on monitor 50. 

To display a panoramic image of facility 30 on monitor 266, subsystem 40 acquires the 
image and renders it to generate a picture as shown in Fig. 13, 14, 15 or 16, in a standard video 
format, for example, NTSC or VGA. This picture is conveyed over link 260 to facility 262 and 
displayed on monitor 266 substantially without additional processing at the receiving end. 
Preferably, subsystem 40 also generates a graphic user interface, including on-screen menus and 
mouse-responsive controls, for example, as are known in the art, which is superimposed on the 
picture conveyed to facility 262. When a user in facility 262 interacts with the interface, the 
interaction invokes a predetermined command message, which is generated by processor 264 
and conveyed over link 260 to subsystem 40. Thus, for example, if the user selects a particular 
region in the picture, for example, by pointing at the region using a mouse, arrow keys or other 
pointing device, this action will cause processor 264 to send a corresponding message over link 
260. The message will be interpreted by subsystem 40 as a command to rotate the panoramic 
image, as described above, so that the selected region is centered in the picture. 

Users at facility 262 may also choose to view individual partial images captured by 
sensor heads 92, rather than viewing the full panoramic image of facility 30. 

Fig. 18C shows still another teleconferencing configuration, in accordance with an 
alternative embodiment of the present invention, in which facility 262 is in communication via 
link 260 with another facility 280. Neither of the facilities is equipped with an image 
acquisition device like ICD 54, as described above, but both have the capability of acquiring 
panoramic images, using methods known in the art. In the case of faeiiity^i'62, the panoramic 
image is acquired by panning camera 268, as described above. Facility 280 may include a 
panoramic imaging camera 282, for example, including a fisheye lens or other suitable optics 
known in the art. Facility 280 also includes a processor 284 and monitor 286, as are known in 
the art. 
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In the configuration of Fig. 1 8C, the principles of the present invention are applied in 
programming processors 264 and/or 284. The processors map the images captured by cameras 
268 and/or 282 onto an annular strip, preferably a cylindrical strip corresponding to a 360^ 
azimuthal view, as described above with reference to Fig. 13. In the case of camera 282, 
5 geometrical distortions due to the optics must also be corrected. The processors then render 
the mapped images to monitors 266 and 286, preferably in the form shown in Fig. 13, 14, 15 or 
16. 

STEREOSCOPIC PANORAMIC IMAGING 
Fig. 19 is a schematic illustration of a panoramic, stereoscopic image capture device 

10 (ICD) 290, in accordance with another preferred embodiment of the present invention. ICD 
290 is based on multifaceted mirror 100, substantially as described above, comprising multiple 
facets 102, For each facet, there is a pair of sensor heads, comprising a left sensor head 292 
and a right sensor head 294, which capture partial images having a substantially common field 
of view, but seen fi-om mutually offset points of view. The points of view of the left and right 

15 sensor heads are offset by a distance D such that if the two partial images for any pair of sensor 
heads are input to a suitable stereoscopic display system, as is known in the art, the partial 
images will fuse to give a pseudo-three-dimensional image. Preferably, all of left sensor heads 
292 share a common left entrance pupil, and all of right sensor heads 294 share a common right 
entrance pupil, as described above with reference to Fig. 4A. 

20 Other placements of the camera heads may also be used. For example, a beamsplitter 

opposite each of facets 102 may be used to split the image between the left and right sensor 
heads. Moreover, in some cases, the sensor heads corresponding to different ones of facets 102 
will be positioned differently, so as to give a wide-field panoramic image when seen from a 
particular point of view. 

25 The respective pairs of partial images captured by the pairs of sensor heads 292 and 294 

are input to an image capture subsystem, like subsystem 40 described above. The subsystem 
combines respective partial images captured by the plurality of left sensor heads 292 to form a 
left panoramic image, and similariy, combines the partial images from right sensor heads 294 to 
form a right panoramic image. The left and right panoramic images are then displayed, using 

30 any suitable stereoscopic image display system known in the art, whereby a pseudo-three- 
dimensional panoramic image is seen. Preferably, the left and right panoramic images are 
mapped and rendered to the display in a form similar to that shown in Fig. 13, 14, 15 or 16 
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above. In this case, front hemiannulus 239 and rear hemiannulus 237 are preferably 
stereoscopically offset, to give a suitable illusion of depth. 

Although preferred embodiments are shown and described above with reference to 
video teleconferencing systems, it will be appreciated that the principles of the present invention 
may similarly be applied to panoramic image capture and display systems for other applications. 
While the preferred embodiments relate generally to overall systems, including various elements 
working in cooperation, it will be clear to those skilled in the art that many of these elements 
and combinations thereof may be used by themselves or in other systems, and that such use is 
within the scope of the present invention. 

It will further be appreciated that the preferred embodiments described above are cited 
by way of example, and the full scope of the invention is limited only by the claims. 
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CLAIMS 



1 . Apparatus for producing a panoramic image, comprising: 

a plurality of video sensor heads, each sensor head generating video signals 
corresponding to a panial image having a respective field of view, such that at least some of the 
5 fields of view include regions of substantial overlap with respective neighboring fields of view, 
and a union of the fields of view substantially covers a scene having an angular extent beyond 
the field of view of any one of the sensor heads; and 

a processor, which receives and combines the partial images fi-om the sensor heads to 
produce a panoramic image of the scene. 

10 2. Apparatus according to claim 1, wherein the panoramic image covers an azimuthal 
angle of at least 90°. 

3. Apparatus according to claim 2, wherein the panoramic image covers an azimuthal 
angle of at least 180° 

4. Apparatus according to claim 3, wherein the panoramic image covers an azimuthal 
1 5 angle substantially equal to 360°. 

5. Apparatus according to any of the preceding claims, wherein the panoramic image has 
an aspect ratio of width to height of at least 2: 1. 

6. Apparatus according to any of the preceding claims, wherein the sensor heads are 
substantially stationary. 

20 7. Apparatus according to any of the preceding claims, wherein each of the plurality of 
sensor heads has a respective entrance pupil, and wherein the sensor heads are so arranged that 
the centers of the entrance pupils of at least some of the plurality of sensor heads substantially 
coincide. 

8. Apparatus according to any of the preceding claims, and comprising beam-combining 
25 optics, which comprise a plurality of mutually-angled reflective surfaces. 

9. Apparatus according to claim 8, wherein the beam combining optics comprise a 
multifaceted mirror, formed by the plurality of mutually-angled reflective surfaces. 

10. Apparatus according to claim 8 or 9, wherein the plurality of mutually-angled surfaces 
comprise planar surfaces. 
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11. Apparatus according to claim 8 or 9, wherein the plurality of mutually-angled surfaces 
comprise cun'ed surfaces. 

12. Apparatus according lo any of claims 8-11, wherein the optics comprise one or more 
baf?les for blocking undesired reflections from the plurality of reflective surfaces. 

5 13. Apparatus according to any of claims 8-12, wherein the optics introduce vignetting in 
the regions of substantial overlap. 

14. Apparatus according to claim 13, wherein the optics are characterized by an intensity 
transfer function with respect to each of the fields of view, such that for each point in at least 
one of the regions of substantial overlap, the sum of the intensity transfer functions with respect 

10 to the two fields of view that overlap there is approximately equal to the value of one of the 
functions at a point in the center of one of the fields of view. 

15. Apparatus according to any of claims 8-14, wherein the optics comprise a generally 
cylindrical refractive element which focuses light onto the plurality of reflective surfaces. 

16. Apparatus according to claim 15, wherein the apparatus comprises a visor generally 
1 5 surrounding the plurality of video sensor heads, and wherein the visor comprises the generally 

cylindrical refractive element. 

17. Apparatus according to any of claims 8-16, wherein the plurality of sensor heads 
comprises a plurality of pairs of sensor heads, each pair associated with a common, respective 
one of the plurality of respective surfaces, and wherein the panoramic image comprises a 

20 stereoscopic image. 

18. Apparatus according to any of the preceding cl^ms, wherein the panoramic image is 
mapped onto a generally cylindrical geometrical image surface. 

19. Apparatus according to claim 18, wherein the processor renders the image surface to 
display the panoramic image in a generally elliptical form on a video screen. 

25 20. Apparatus according to claim 19, wherein the processor is controlled to alter a point of 
view from which the image surface is rendered. 

21. Apparatus according to claim 19 or 20, wherein the processor is controlled so as to 
geometrically rotate the image surface. 
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22. Apparatus according to any of claims 19-21, and comprising a user input device, 
wherein the processor renders the image responsive to an input from the user input device. 

23. Apparatus according to claim 22, wherein the user input device comprises a pointing 
device, for indicating a region of interest in the image. 

5 24. Apparatus according to any of the preceding claims, wherein the processor applies 
timing signals to the sensor heads so as to commonly synchronize at least some of the pluraHty 
of sensor heads. 

25. Apparatus according to claim 24, wherein the processor compresses the panoramic 
image, and wherein the processor controls the timing signals so as to enhance the image 

1 0 compression. 

26. Apparatus according to any of the preceding claims, wherein in combining the partial 
images, the processor remaps the images according to a magnification that varies as a function 
of azimuthal angle, 

27. Apparatus according to claim 26, wherein the processor remaps the images so as to 
1 5 correct for perspective variations in the images. 

28. Apparatus according to claim 27, wherein the processor remaps the images so that 
every one of a group of figures in the panoramic image appears to be at approximately the same 
distance from the apparatus capturing the image. 

29. Apparatus according to any of the preceding claims, wherein the processor suppresses a 
20 portion of the panoramic image. 

30. Apparatus according to any of the preceding claims, wherein the processor variably 
controls a parameter of the panoramic image as a function of azimuthal angle. 

3 1 . Apparatus according to claim 30, wherein the processor applies an azimuthally-variable 
gain to the video signals. * \ , 

25 32. Apparatus according to claim 30 or 31, wherein the processor adjiijsts the brightness of 
the panoramic image by a factor that varies as a function of azimuthal angle. 

33. Apparatus according to any of claims 30-32, wherein the processor adjusts the contrast 
of the panoramic image by applying a correction to the image that varies as a function of 
azimuthal angle. 
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34. Apparatus according to any of claims 30-33, wherein the processor corrects the video 
signals so that the panoramic image has a substantially uniform color balance, independent of 
azimuthal angle. 

35. Apparatus according to any of claims 30-34, wherein the processor applies an image 
5 processing operation to the video signals, as a function of azimuthal angle, so that the 

panoramic image has a substantially enhanced dynamic range. 

36. Apparatus according to any of the preceding claims, wherein the panoramic image 
comprises a reference image, and wherein the processor updates a portion of the reference 
image responsive to a video signal received from a selected one or more of the plurality of 

1 0 sensor heads. 

37. Apparatus according to any of the preceding claims, and comprising a microphone 
array, which receives sound from the scene and generates audio signals, and an audio 
processor, which processes the audio signals responsive to a direction corresponding to a 
selected region in the scene. 

15 38. Apparatus according to claim 37, wherein a viewer of the panoramic image selects the 
region by indicating a point in the panoramic image. 

39. Apparatus according to claim 37 or 38, wherein the audio signals are processed to 
enhance the sound received from the direction. 

40. Apparatus according to claim 37 or 38, wherein the audio signals are processed to mute 
20 the sound reaching the array from the direction. 

41. Apparatus according to any of claims 37-40, wherein the processor receives the audio 
signals from the microphone array and selects one or more of the plurality of sensor heads to 
select based on the audio signals. 

42. Apparatus for producing a panoramic image, comprising; 

25 an image capture device, which generates video signals responsive to a panoramic 

scene; and 

a processor, which receives the video signals and applies a processing operation thereto, 
which varies as a fijnction of azimuthal angle, to produce a panoramic image of the scene. 

43. Apparatus according to claim 42, wherein the processing operation comprises an image 

30 magnification, varj'ing as a function of the azimuthal angle. 
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44. Apparatus according to claim 43, wherein the processing operation varies the 
magnification to adjust for differences in the distances of objects in the scene from the image 
capture device. 

45. Apparatus according to any of claims 42-44, wherein the processing operation 
5 comprises an adjustment of the image brightness as a function of the azimuthal angle. 

46. Apparatus according to any of claims 42-45, wherein the processing operation 
comprises an adjustment of the image contrast as a function of the azimuthal angle. 

47. Apparatus according to any of claims 42-46, wherein the processing operation 
comprises an adjustment of the image color balance as a function of the azimuthal angle. 

10 48. Apparatus according to any of claims 42-47, wherein the processing operation 
comprises an enhancement of the image dynamic range as a function of the azimuthal angle. 

49. Apparatus for image compression, comprising: 

at least one video sensor head, which captures an image of a scene, and 
a controller, which receives, digitizes and compresses the image, and which exchanges 
15 control signals with the at least one video sensor head so as to reduce the data size of the 
compressed image. 

50. Apparatus according to claim 49, wherein the controller provides timing signals to the 
at least one sensor head. 

51. Apparatus according to claim 49-50, wherein the at least one sensor head comprises a 
20 plurality of sensor heads, which receive respective partial images, and wherein the controller 

controls the plurality of sensor heads and combines the partial images received therefrom to 
produce a panoramic image. 

52. Apparatus according to claim 51, wherein the controller selects an image sector within 
the panoramic image and receives partial images corresponding to the image sector from one or 

25 more of the plurality of sensor heads so as to update the panoramic image. 

53. Apparatus according to any of claims 49-52, wherein the controller identifies an area of 
the panoramic image corresponding to a generally static background region of the scene, so as 
to update the area less frequently than other areas of the image. 
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54. Apparatus according to any of claims 49-53, wherein the controller receives a video 
sensor parameter from the at least one sensor head and transmits the parameter to a remote 
site. 

55. Apparatus according to claim 54, wherein the sensor parameter comprises a gain factor. 

5 56. Apparatus according to claim 54 or 55, wherein the sensor parameter comprises a 
shutter speed. 

57. Apparatus according to any of claims 54-56, wherein the sensor parameter comprises a 
color balance setting. 

58. Apparatus for displaying a panoramic image, comprising: 
10 a video display; and 

a processor, which receives panoramic image data, geometrically maps the data onto a 
generally annular surface, and renders at least a portion of the surface to the display. 

59. Apparatus according to claim 58, wherein the processor renders the generally annular 
surface in the form of two hemiannular strips. 

15 60. Apparatus according to claim 59, wherein the processor performs a left-right reverse 

operation on a portion of the image on one of the hemiannular strips. 

61. Apparatus according to any of claims 58-60, wherein the generally annular surface 
comprises a generally cylindrical surface. 

62. Apparatus according to claim 61, wherein the processor renders the generally cylindrical 
20 surface to the display in a generally elliptical form. 

63. Apparatus according to any of claims 58-62, wherein the processor varies a point of 
view from which the surface is rendered. 

64. Apparatus according claim 58, wherein the video display comprises a generally annular 
display screen. 

25 65. Apparatus according to claim 64, wherein the display screen comprises a back- 
projection screen, and comprising a video projector, which projects the image onto an inner 
side of the screen, so that the image is visible from the outer side thereof 
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66. Apparatus according to claim 65, and comprising a partially-reflecting mirror, which 
directs the image projected by the projector onto the back- projection screen, so that the image 
on the screen can be seen through the mirror. 

67. Apparatus according to any of claims 58-66, wherein the processor geometrically 
rotates the annular surface which is rendered. 

68. Apparatus according to claim 67, and comprising a user interface, wherein a viewer of 
the display operates the user interface to alter the rendition of the surface. 

69. Apparatus according to claim 68, wherein the user interface comprises a pointing 
device, which the viewer uses to indicate a desired point with respect to which the surface is to 
be rotated. 

70. Apparatus according to claim 68 or 69, wherein the processor remaps at least a portion 
of the data, responsive to an input from the user interface, so as to zoom in on a region of 
interest indicated by the viewer. 

71 . Apparatus according to any of claims 58-70, wherein the processor renders a picture-in- 
picture image, which is inset in the panoramic image on the display. 

72. A method for displaying a panoramic image on a video display, comprising: 
defining an annular strip and a point of view outside a volume bounded by the strip; 
mapping the panoramic image onto the annular strip; and 

rendering an image of the strip, as seen from the point of view, to the video display. 

73. A method according to claim 72, wherein mapping the panoramic image onto the 
annular strip comprises: 

mapping a first segment of the image, in relative proximity to the point of view, onto a 
near hemiannulus; 

mapping a second segment of the image, relatively distant' from the point of view, onto 
a far hemiannulus; and 

reversing at least a portion of the first segment. 

74. A method according to claim 73, wherein reversing at least a portion of the first 
segment comprises reversing left-right orientation of a figure in the segment. 

75. A method according to any of claims 72-74, wherein defining the annular strip 
comprises defining a generally cylindrical strip, and defining the point of view comprises 
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defining a point of view above an equatorial plane of the annular strip, so that the image of the 
strip rendered to the display is generally elliptical in shape. 

76. A method according to any of claims 72-75 and comprising rotating the annular strip 
about an axis thereof, relative to the point of view, so as to shift the image rendered of the strip 

5 as seen from the point of view. 

77. A method according to claim 76, wherein rotating the strip comprises altering the image 
of the strip substantially without engendering mechanical motion. 

78. A method according to claim 76 or 77, and comprising selecting a region of interest in 
the image, wherein rotating the strip comprises rotating the strip by such an angle that the 

10 selected region is brought to a point opposite the point of view. 

79. A method according to claim 78, and comprising receiving audio signals from an area 
shown in the panoramic image, wherein selecting the region of interest comprises identifying a 
direction from which the audio signals are received and choosing a region corresponding to the 
direction. 

15 80. A method according to claim 79, wherein receiving the audio signals comprises 
receiving signals from a acoustic signaling device. 

81. A method according to any of claims 78-80, and comprising displaying an enlarged 
image of the selected region on the video display. 

82. A method according to any of claims 72-77, and comprising displaying an enlarged 
20 image of a selected region on the video display. 

83. A method according to claim 81 or 82, wherein displaying the enlarged image 
comprises displaying the image alongside the image of the strip. 

84. A method according to claim 81 or 82, wherein displaying the enlarged image 
comprises mapping the panoramic image so as to enlarge a portion of theyannular strip that 

25 includes the selected region. 

85. A method according to any of claims 8 1 -84, " wherein displaying the enlarged image 
comprises enhancing the resolution of the image in the selected region. 

86. A method for displaying a panoramic image, comprising: 



providing a generally annular back-projection screen; 
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mapping and rendering the panoramic image to the screen; 
projecting the image onto the screen from inside the annulus; and 
viewing the image from outside the annulus. 

87. A method according to claim 86, wherein projecting the image comprises projecting a 
5 video image. 

88. A method according to claim 86 or 87, wherein projecting the image comprises 
projecting the image against a partially-reflecting mirror, which reflects the image onto the 
screen, and wherein viewing the image comprises viewing a portion of the screen by way of the 
mirror. 

10 89. A method according to any of claims 72-77, 82, or 86-88, and comprising: 

receiving audio signals from an array of microphones placed adjacent to an area shown 
in the panoramic image; 

selecting a region of interest in the image; and 

processing the audio signals selectively, responsive to the selected region. 

15 90. A method according to claim 89, wherein processing the signals selectively comprises 
reconstructing sounds originating from a vicinity of the region in the area shown in the 
panoramic image. 

91. A method according to claim 89, wherein processing the signals selectively comprises 
muting sounds originating from a vicinity of the region in the area shown in the panoramic 

20 image. 

92. A method according to any of claims 72-91, wherein mapping and rendering the image 
comprise mapping and rendering a reference image, updating a selected portion of the image, 
and rendering the selected, updated portion. 

93. A method according to claim 92, and comprising producing the reference panoramic 
25 image by combining a plurality of images respectively captured by a plurality of video sensor 

heads, wherein updating the selected portion comprises updating a portion of the image 
captured by a selected one or more of the plurality of video sensor heads. 

94. A method according to claim 93, and comprising selecting the one or more of the 
plurality of video sensor heads by indicating an area on the video display corresponding to the 

30 portion of the image captured by the one or more sensor heads. 
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95. A method according to any of claims 72-92, and comprising receiving a plurality of 
images from a remote location, the plurality of video images covering a panoramic field of view 
at the location, and generating the panoramic image by combining the video images. 

96. A method for transmitting an image to a remote site, comprising: 
5 generating video signals using one or more video sensor heads; 

receiving and compressing the video signals for transmission; and 
exchanging control signals with the one or more video sensor heads so as to reduce the 
data size of the compressed signals. 

97. A method according to claim 96, wherein exchanging the control signals comprises 
10 receiving a sensor head control parameter and transmitting the parameter to the remote site. 

98. A method according to claim 97, wherein receiving the control parameter comprises 
receiving a gain factor. 

99. A method according to claim 97 or 98, wherein recei\dng the control parameter 
comprises receiving an electronic shutter setting. 

15 100. A method according to any of claims 97-99, wherein receiving the control parameter 
comprises receiving a color balance setting. 

101. A method according to any of claims 97-100, wherein transmitting the parameter 
comprises transmitting a change in a parameter relative to an earlier value thereof 

102. A method according to any of claims 96-101, wherein generating the video signals 
20 comprises generating signals using a plurality of video sensor heads, and wherein exchanging 

the control signals comprises selecting one or more of the plurality of sensor heads to transmit 
signals for use in updating a corresponding portion of a reference image, while other portions 
of the image corresponding to the non-selected sensor heads are substantially frozen. 

103. A method for producing a panoramic image of a scene, comprising: 

25 positioning a plurality of video sensor heads to capture a respective plurality of partial 

images of the scene, each partial image having an angular extent substantially less than a total 
angular extent of the scene whose image is to be pfoduced and having a region of overlap with 
a neighboring partial image; 

receiving video partial images from the plurality of sensor heads; and 
30 combining the partial images to produce the panoramic image. 
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104. A method according to claim 103, wherein producing the panoramic image comprises 
producing an image covering an azimuthal field of view of at least 90^. 

105. A method according to claim 104, wherein producing the image comprises producing an 
image covering an azimuthal field of view of at least 180®. 

5 106. A method according to claim 105, wherein producing the image comprises producing an 
image covering an azimuthal field of view substantially equal to 360^. 

107. A method according to any of claims 103-106, wherein producing the panoramic image 
comprises producing an image having an aspect ratio of width to height of at least 2: 1. 

108. A method according to any of claims 103-107, wherein positioning the sensor heads 
10 comprising positioning the sensor heads so as to be substantially stationary, 

109. A method according to any of claims 103-108, wherein positioning the sensor heads 
comprises aligning at least some of the sensor heads and optics associated therewith, each such 
sensor head having a respective entrance pupil, so that the centers of the entrance pupils 
substantially coincide. 

15 110. A method according to any of claims 103-109, wherein combining the partial images 
comprises adding each partial image to its neighboring image in the overlap region thereof 

111. A method according to any of claims 103-110, wherein combining the partial images to 
produce the panoramic image comprises mapping the images onto a substantially cylindrical 
image surface. 

20 112. A method according to claim 111, and comprising rendering the panoramic image to a 
video display in a generally elliptical form. 

113. A method according to claim III, and comprising rendering and projecting the 
panoramic image onto a generally annular back-projection display. 

114. A method according to any of claims 103-113, wherein combining the partial images 
25 comprises automatically varying a display mode of the combined images responsive to a 

predetermined condition in the scene. 

115. A method according to any of claims 103-114, wherein positioning the plurality of 
sensor heads comprises positioning a plurality of pairs of sensor heads, the pair comprising left 
and right sensor heads, mutually offset, so as to capture two partial images of a respective, 
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common field of view, and wherein combining the partial images comprises combining the 
images to produce a stereoscopic panoramic image. 

116, A method according to any of claims 103-1 15, wherein combining the images comprises 
processing the images variably as a function of azimuthal angle in the panoramic image. 

5 117. A method for producing a panoramic image comprising: 

receiving video images covering a panoramic field of view; and 

processing the images variably as a fiinction of azimuthal angle in the panoramic image. 

118. A method according to claim 116 or 117, wherein processing the images comprises 
applying a variable gain to video signals corresponding to the images, as a fiinction of azimuthal 

10 angle. 

119. A method according to any of claims 116-118, wherein processing the images 
comprises adjusting an image brightness as a fiinction of azimuthal angle. 

120. A method according to any of claims 116-119, wherein processing the images 
comprises adjusting an image contrast as a function of azimuthal angle. 

15 121. A method according to any of claims 116-120, wherein processing the images 
comprises adjusting a color balance as a fijnction of azimuthal angle. 

122. -A method according to any of claims 116-121, wherein processing the images 
comprises adjusting an image magnification as a function of azimuthal angle. 

123. A method according to claim 122, wherein adjusting the magnification comprises 
20 correcting the image for perspective differences among a plurality of objects seen in the 

panoramic image. 

124. A method according to any of claims 116-123, wherein processing the images 
comprises suppressing a portion of the images falling within a selected range of azimuthal 
angles. 

25 125. A system for video teleconferencing, comprising at least one transmitting station, which 



a plurality of video sensor heads, positionable at or near the center of a table around 
which teleconferencing participants are seated so as to capture images of the participants on 
opposing sides of the table; and 
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a transmission processor, which receives and combines video signals from the sensor 
heads to produce and transmit a panoramic image of the scene. 

126. A system according to claim 125, wherein the plurality of sensor heads capture images 
of the participants along generally horizontal respective optical axes. 

5 127. A system according to claim 125 or 126, wherein the transmitting station comprises a 
microphone array adjacent to the plurality of sensor heads. 

128. A system according to any of claims 125-127, wherein each video sensor head 
generates video signals corresponding to a partial image having a respective field of view, such 
that at least some of the fields of view include regions of substantial overlap with respective 

10 neighboring fields of view, and a union of the fields of view substantially covers a scene having 
an angular extent beyond the field of view of any one of the sensor heads. 

129. A system according to any of claims 125-128, and comprising at least one receiving 
station, which comprises: 

a video monitor; 

15 a receiving processor, which receives the transmitted panoramic image and displays the 

image to the monitor; and 

user controls, which provide user commands to both the receiving processor and the 
transmission processor. 

130. A system according to claim 129, wherein the panoramic image is mapped onto a 
20 generally cylindrical geometrical image surface. 

131. A system according to claim 130, wherein the panoramic image is mapped by the 
transmission processor. 

132. A system according to claim 130, wherein the panoramic image is mapped by the 
receiving processor. 

25 133. A system according to any of claims 130-132, wherein the surface is rendered to the 
display in a generally elliptical form. * - 

134. .A system according to claim 133, wherein the surface is rendered by the transmission 
processor. 
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135. A system according to claim 133, wherein the surface is rendered by the receiving 
processor. 

136. A system according to any of claims 130-135, wherein the user controls generate a 
geometrical rotation of the image surface. 

5 137. A system according to any of claims 130-136, wherein the user controls determine a 
point of view from which the image surface is rendered. 

138. A system according to any of claims 129-137, wherein the user controls of the receiving 
station select one or more of the plurality of sensor heads of the transmitting station, and 
wherein the video display is updated using images captured by the one or more selected sensor 

10 heads. 

139. A system according to any of claims 129-138, wherein the transmitting station 
comprises an array of microphones, which generate respective audio signals, and wherein the 
receiving processor receives the audio signals and processes them to reconstruct sound 
originating in a vicinity of a point in the panoramic image selected using the user controls. 

15 140. A system according to any of claims 129-139, wherein the user controls of the receiving 
station comprise a menu generated by the transmitting station and displayed on the video 
display of the receiving station. 

141. A system according to any of claims 129-140, wherein the transmitting station 
comprises user controls capable of performing at least some of the functions of the user 

20 controls of the receiving station. 

142. A system according to any of claims 129-141, wherein the transmission processor 
exchanges sensor control parameters with the plurality of video sensor heads and transmits at 
least some of the parameters to the receiving processor, for use in rendering the image. 

143. A system according to claim 142, wherein the transmission processor compresses the 
25 image for transmission, and wherein the receiving processor decompresses the image using the 

transmitted parameters. 

144. Apparatus for producing a stereoscopic panoramic image, comprising: 

a plurality of video sensor heads, which capture respective partial images of a scene; 

and 
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a processor, which receives and combines the partial images from the sensor heads tc 
produce a stereoscopic panoramic image of the scene. 

145. Apparatus according to claim 144, wherein the plurality of video sensor heads are 
substantially stationary'. 

5 146. Apparatus according to claim 144 or 145, wherein the panoramic image covers a field 
of view having an angular extent substantially greater than that of the field of view of any one 
of the plurality of sensor heads. 

147. Apparatus according to any of claims 144-146, wherein the panoramic image covers an 
azimuthal angle of at least 90°. 

10 148. Apparatus according to claim 147, wherein the panoramic image covers an azimuthal 
angle of at least 180^ 

149. Apparatus according to claim 148, wherein the panoramic image covers an azimuthal 
angle substantially equal to 360° 

150. Apparatus according to any of claims 144-149, wherein the panoramic image has an 
1 5 aspect ratio of width to height of at least 2:1. 

151. Apparatus according to any of claims 144-150, wherein the plurality of video sensor 
heads comprises a plurality of pairs of video sensor heads, each pair comprising mutually offset 
left and right sensor heads, which capture respective left and right partial images of a 
substantially common field of view. 

20 152. Apparatus according to claim 151, wherein the processor combines the partial images to 
form left and right panoramic images, which are viewed jointly to produce the stereoscopic 
panoramic image. 

153. Apparatus according to claim 151 or 152, wherein each of the plurality of sensor heads 
has a respective entrance pupil, and wherein the sensor heads are so arranged that the centers of 
25 the entrance pupils of at least some of the left sensor heads substantially mutually coincide, and 
the centers of the entrance pupils of at least some of the right sensor heads substantially 
mutually coincide 
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154. Apparatus according to any of claims 151-153, and comprising beam-combining optics, 
which comprise a plurality of substantially planar, mutually-angled reflective surfaces, wherein 
one of the surfaces is associated with each of the plurality of pairs of sensor heads. 

155. Apparatus according to claim 154, wherein the beam combining optics comprise a 
multifaceted mirror, formed by the plurality of mutually-angled reflective surfaces. 
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FURTHER INFORMATION CONTINUED FROM PCT/ISA/ 21Q 



1. Claims: 1-41, 103-116 

Method and apparatus for producing a panoramic image, 
comprising a plurality of video cameras, each camera 
generating video signals corresponding to a partial image 
having a respective field of view, at least some of the 
fields of view include regions of substantial overlap with 
respective neihgbouring fields of view, and the combination 
of all the fields of view substantially covering a scene 
having an angular extent beyond the. field of view of any one 
of the cameras. 

The partial images of the cameras are received by a 
processor, which produces a panoramic image of the scene. 



2. Claims: 42-48, 117-124 

Method and apparatus for producing a panoramic image, 
comprising an image capture device, generating video signals 
responsive to a panoramic scene, and a processor, which 
receives the video signals and produces a panoramic image of 
the scene by applying a processing operation thereto, which 
varies as a function of azimuthal angle. 



3. Claims: 49-57, 96-102 

Method and apparatus for compressing image signals, 
comprising a controller, which exchanges control signals 
with one or more video cameras, delivering the image 
signals, in order to reduce the data size of the compressed 
image signals. 



4. Claims: 58-95 

Method and apparatus for displaying a panoramic image, by 
geometrically mapping the image data onto a generally 
annular surface and by rendering at least a portion of the 
surface to a display. 



5. Claims: 125-143 

Video teleconferencing system comprising a transmitting 
station having a plurality of video cameras, positionable at 
or near the center of a conference table, in order to 
capture images of the participants of the conference, and a 
transmission processor, which receives and combines the 
video signals of the cameras in order to produce and to 
transmit a panoramic image of the conference scene. 



5. Claims: 144-155 
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FURTHER INFORMATION CONTINUED FROM PCT/ISA/ 210 



Apparatus for producing a stereoscopic panoramic image, 
comprising a plurality of video cameras, capturing 
respective partial images of a scene, and a processor, which 
receives the partial images and produces a stereoscopic 
panoramic image of the scene. 
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